
# Installing the libraries with the specified version.
# uncomment and run the following line if Google Colab is being used
#!pip install scikit-learn==1.2.2 seaborn==0.13.1 matplotlib==3.7.1 numpy==1.25.2 pandas==1.5.3 yellowbrick==1.5 -q --user


# Import libraries we need to do our work

# Read and manipulate data
import numpy as np
import pandas as pd

# HTML styling
from IPython.display import display, HTML

# Data visualization
import matplotlib.pyplot as plt
import seaborn as sns

# Statistics
from scipy.stats import shapiro

# Suppress warnings
import warnings
warnings.filterwarnings("ignore")


# Installing the libraries with the specified version.
# uncomment and run the following lines if Jupyter Notebook is being used
#!pip install scikit-learn==1.2.2 seaborn==0.13.1 matplotlib==3.7.1 numpy==1.25.2 pandas==1.5.2 yellowbrick==1.5 -q --user
#!pip install --upgrade -q jinja2


# Load Google Colab
from google.colab import drive
drive.mount('/content/drive')

Mounted at /content/drive


# Define path to data file
path = r'/content/drive/MyDrive/Learning/Data Coursework/PGP-DSBA/7-Unsupervised Learning/Project 7/'
df = pd.read_csv(path + 'stock_data.csv')


# Iterate through each column and determine its data type

# Calculate number of rows and columns
num_rows, num_cols = df.shape

# Initialize dictionary to store counts of each data type
data_type_counts = {}

# Iterate through each column and determine its data type
for col in df.columns:
    data_type = str(df[col].dtype)
    if data_type in data_type_counts:
        data_type_counts[data_type] += 1
    else:
        data_type_counts[data_type] = 1

# # Check size of dataset and print summary information
print(f'The data has {num_rows} rows and {num_cols} columns total:')
print()
for dtype, count in data_type_counts.items():
    print(f'{count} columns of type {dtype}')

The data has 340 rows and 15 columns total:

4 columns of type object
7 columns of type float64
4 columns of type int64


# View information
print('Training')
display(df.info())

Training
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 340 entries, 0 to 339
Data columns (total 15 columns):
 #   Column                        Non-Null Count  Dtype  
---  ------                        --------------  -----  
 0   Ticker Symbol                 340 non-null    object 
 1   Security                      340 non-null    object 
 2   GICS Sector                   340 non-null    object 
 3   GICS Sub Industry             340 non-null    object 
 4   Current Price                 340 non-null    float64
 5   Price Change                  340 non-null    float64
 6   Volatility                    340 non-null    float64
 7   ROE                           340 non-null    int64  
 8   Cash Ratio                    340 non-null    int64  
 9   Net Cash Flow                 340 non-null    int64  
 10  Net Income                    340 non-null    int64  
 11  Earnings Per Share            340 non-null    float64
 12  Estimated Shares Outstanding  340 non-null    float64
 13  P/E Ratio                     340 non-null    float64
 14  P/B Ratio                     340 non-null    float64
dtypes: float64(7), int64(4), object(4)
memory usage: 40.0+ KB

None


# Describe the data. Can also use include=all to also show categorical series.
df.describe(include='all').T


# Check for empty values
print(f'Dataset has {df.isnull().sum().sum()} null values:',)

# Replace any missing values with NaNs
df = df.replace("?", np.nan)

Dataset has 0 null values:


# Retrieve and preview the categorial column values, using a list.
df_eda = df.copy()

# Create numerical and categorical column lists
list_catcol = df_eda.select_dtypes(include=['object']).columns.to_list()
list_numcol = df_eda.select_dtypes(exclude=['object']).columns.to_list()

print('Dataset has', len(list_catcol), 'categorical columns and', len(list_numcol), 'numerical columns.')

Dataset has 4 categorical columns and 11 numerical columns.


# Use a for loop to show the counts and percent contribution of each numerical column.
for col in list_numcol:
  print(f'***** {col} *****')
  print('Preview counts for:',df_eda[col].value_counts(),'\n')
  print('Preview % breakdown for:',df_eda[col].value_counts(normalize=True),'\n')

***** Current Price *****
Preview counts for: Current Price
54.360001     2
114.379997    2
43.230000     2
32.930000     2
103.889999    1
             ..
51.070000     1
52.130001     1
81.589996     1
302.399994    1
47.919998     1
Name: count, Length: 336, dtype: int64 

Preview % breakdown for: Current Price
54.360001     0.005882
114.379997    0.005882
43.230000     0.005882
32.930000     0.005882
103.889999    0.002941
                ...   
51.070000     0.002941
52.130001     0.002941
81.589996     0.002941
302.399994    0.002941
47.919998     0.002941
Name: proportion, Length: 336, dtype: float64 

***** Price Change *****
Preview counts for: Price Change
 9.999995     1
 6.237855     1
 6.585554     1
 5.796884     1
 9.529966     1
             ..
 8.037605     1
 10.019650    1
-4.078593     1
 3.654238     1
 16.678836    1
Name: count, Length: 340, dtype: int64 

Preview % breakdown for: Price Change
 9.999995     0.002941
 6.237855     0.002941
 6.585554     0.002941
 5.796884     0.002941
 9.529966     0.002941
                ...   
 8.037605     0.002941
 10.019650    0.002941
-4.078593     0.002941
 3.654238     0.002941
 16.678836    0.002941
Name: proportion, Length: 340, dtype: float64 

***** Volatility *****
Preview counts for: Volatility
1.687151    1
1.023375    1
1.460619    1
1.281566    1
2.168814    1
           ..
1.056186    1
1.308082    1
1.941104    1
1.404508    1
1.610285    1
Name: count, Length: 340, dtype: int64 

Preview % breakdown for: Volatility
1.687151    0.002941
1.023375    0.002941
1.460619    0.002941
1.281566    0.002941
2.168814    0.002941
              ...   
1.056186    0.002941
1.308082    0.002941
1.941104    0.002941
1.404508    0.002941
1.610285    0.002941
Name: proportion, Length: 340, dtype: float64 

***** ROE *****
Preview counts for: ROE
10     24
9      16
14     16
11     16
17     15
       ..
582     1
116     1
68      1
63      1
142     1
Name: count, Length: 82, dtype: int64 

Preview % breakdown for: ROE
10     0.070588
9      0.047059
14     0.047059
11     0.047059
17     0.044118
         ...   
582    0.002941
116    0.002941
68     0.002941
63     0.002941
142    0.002941
Name: proportion, Length: 82, dtype: float64 

***** Cash Ratio *****
Preview counts for: Cash Ratio
99     37
47     19
25     10
70      8
10      7
       ..
82      1
128     1
175     1
164     1
100     1
Name: count, Length: 124, dtype: int64 

Preview % breakdown for: Cash Ratio
99     0.108824
47     0.055882
25     0.029412
70     0.023529
10     0.020588
         ...   
82     0.002941
128    0.002941
175    0.002941
164    0.002941
100    0.002941
Name: proportion, Length: 124, dtype: float64 

***** Net Cash Flow *****
Preview counts for: Net Cash Flow
 159000000    2
 39000000     2
 160383000    2
-8000000      2
 23000000     2
             ..
 4073000      1
-14756000     1
 523803000    1
 2196000      1
-43623000     1
Name: count, Length: 332, dtype: int64 

Preview % breakdown for: Net Cash Flow
 159000000    0.005882
 39000000     0.005882
 160383000    0.005882
-8000000      0.005882
 23000000     0.005882
                ...   
 4073000      0.002941
-14756000     0.002941
 523803000    0.002941
 2196000      0.002941
-43623000     0.002941
Name: proportion, Length: 332, dtype: float64 

***** Net Income *****
Preview counts for: Net Income
1899000000    2
355000000     2
848000000     2
1034000000    2
636056000     2
             ..
268000000     1
232120000     1
878485000     1
85171000      1
339000000     1
Name: count, Length: 335, dtype: int64 

Preview % breakdown for: Net Income
1899000000    0.005882
355000000     0.005882
848000000     0.005882
1034000000    0.005882
636056000     0.005882
                ...   
268000000     0.002941
232120000     0.002941
878485000     0.002941
85171000      0.002941
339000000     0.002941
Name: proportion, Length: 335, dtype: float64 

***** Earnings Per Share *****
Preview counts for: Earnings Per Share
3.28    5
3.71    4
3.03    4
1.80    4
1.58    4
       ..
5.87    1
1.86    1
1.77    1
1.31    1
1.20    1
Name: count, Length: 268, dtype: int64 

Preview % breakdown for: Earnings Per Share
3.28    0.014706
3.71    0.011765
3.03    0.011765
1.80    0.011765
1.58    0.011765
          ...   
5.87    0.002941
1.86    0.002941
1.77    0.002941
1.31    0.002941
1.20    0.002941
Name: proportion, Length: 268, dtype: float64 

***** Estimated Shares Outstanding *****
Preview counts for: Estimated Shares Outstanding
1.992378e+08    4
5.139877e+08    3
1.115226e+08    3
1.469542e+09    2
1.633016e+09    2
               ..
3.671392e+08    1
5.777662e+07    1
5.457799e+08    1
1.485114e+08    1
4.985294e+08    1
Name: count, Length: 323, dtype: int64 

Preview % breakdown for: Estimated Shares Outstanding
1.992378e+08    0.011765
5.139877e+08    0.008824
1.115226e+08    0.008824
1.469542e+09    0.005882
1.633016e+09    0.005882
                  ...   
3.671392e+08    0.002941
5.777662e+07    0.002941
5.457799e+08    0.002941
1.485114e+08    0.002941
4.985294e+08    0.002941
Name: proportion, Length: 323, dtype: float64 

***** P/E Ratio *****
Preview counts for: P/E Ratio
93.089287    17
20.819876     6
24.070121     5
28.407929     3
17.313076     3
             ..
18.636363     1
21.175469     1
55.829113     1
7.575269      1
70.470585     1
Name: count, Length: 295, dtype: int64 

Preview % breakdown for: P/E Ratio
93.089287    0.050000
20.819876    0.017647
24.070121    0.014706
28.407929    0.008824
17.313076    0.008824
               ...   
18.636363    0.002941
21.175469    0.002941
55.829113    0.002941
7.575269     0.002941
70.470585    0.002941
Name: proportion, Length: 295, dtype: float64 

***** P/B Ratio *****
Preview counts for: P/B Ratio
-1.269332     6
-3.089477     5
-3.973395     4
 3.954975     4
-4.327138     4
             ..
-14.151445    1
 5.108756     1
 4.427425     1
 5.884467     1
 1.723068     1
Name: count, Length: 299, dtype: int64 

Preview % breakdown for: P/B Ratio
-1.269332     0.017647
-3.089477     0.014706
-3.973395     0.011765
 3.954975     0.011765
-4.327138     0.011765
                ...   
-14.151445    0.002941
 5.108756     0.002941
 4.427425     0.002941
 5.884467     0.002941
 1.723068     0.002941
Name: proportion, Length: 299, dtype: float64


# Use a for loop to show the counts and percent contribution of each categorical column.
for col in list_catcol:
  print(f'***** {col} *****')
  print('Preview counts for:',df_eda[col].value_counts(),'\n')
  print('Preview % breakdown for:',df_eda[col].value_counts(normalize=True),'\n')

***** Ticker Symbol *****
Preview counts for: Ticker Symbol
AAL     1
NEE     1
NUE     1
NTRS    1
NSC     1
       ..
EQR     1
EQIX    1
EOG     1
EMN     1
ZTS     1
Name: count, Length: 340, dtype: int64 

Preview % breakdown for: Ticker Symbol
AAL     0.002941
NEE     0.002941
NUE     0.002941
NTRS    0.002941
NSC     0.002941
          ...   
EQR     0.002941
EQIX    0.002941
EOG     0.002941
EMN     0.002941
ZTS     0.002941
Name: proportion, Length: 340, dtype: float64 

***** Security *****
Preview counts for: Security
American Airlines Group    1
NextEra Energy             1
Nucor Corp.                1
Northern Trust Corp.       1
Norfolk Southern Corp.     1
                          ..
Equity Residential         1
Equinix                    1
EOG Resources              1
Eastman Chemical           1
Zoetis                     1
Name: count, Length: 340, dtype: int64 

Preview % breakdown for: Security
American Airlines Group    0.002941
NextEra Energy             0.002941
Nucor Corp.                0.002941
Northern Trust Corp.       0.002941
Norfolk Southern Corp.     0.002941
                             ...   
Equity Residential         0.002941
Equinix                    0.002941
EOG Resources              0.002941
Eastman Chemical           0.002941
Zoetis                     0.002941
Name: proportion, Length: 340, dtype: float64 

***** GICS Sector *****
Preview counts for: GICS Sector
Industrials                    53
Financials                     49
Health Care                    40
Consumer Discretionary         40
Information Technology         33
Energy                         30
Real Estate                    27
Utilities                      24
Materials                      20
Consumer Staples               19
Telecommunications Services     5
Name: count, dtype: int64 

Preview % breakdown for: GICS Sector
Industrials                    0.155882
Financials                     0.144118
Health Care                    0.117647
Consumer Discretionary         0.117647
Information Technology         0.097059
Energy                         0.088235
Real Estate                    0.079412
Utilities                      0.070588
Materials                      0.058824
Consumer Staples               0.055882
Telecommunications Services    0.014706
Name: proportion, dtype: float64 

***** GICS Sub Industry *****
Preview counts for: GICS Sub Industry
Oil & Gas Exploration & Production            16
REITs                                         14
Industrial Conglomerates                      14
Electric Utilities                            12
Internet Software & Services                  12
                                              ..
Technology Hardware, Storage & Peripherals     1
Real Estate Services                           1
Trucking                                       1
Networking Equipment                           1
Casinos & Gaming                               1
Name: count, Length: 104, dtype: int64 

Preview % breakdown for: GICS Sub Industry
Oil & Gas Exploration & Production            0.047059
REITs                                         0.041176
Industrial Conglomerates                      0.041176
Electric Utilities                            0.035294
Internet Software & Services                  0.035294
                                                ...   
Technology Hardware, Storage & Peripherals    0.002941
Real Estate Services                          0.002941
Trucking                                      0.002941
Networking Equipment                          0.002941
Casinos & Gaming                              0.002941
Name: proportion, Length: 104, dtype: float64


# Check counts, unique counts and duplicate values for each column. Based on this, we can decide how to approach each variable.
for col in df_eda:
  unique_count = df_eda[col].nunique()
  duplicate_count = df_eda[col].count()-df_eda[col].nunique()
  empty_count = df_eda[col].isnull().sum().sum()
  print(f'Series {col} has {unique_count} unique values, {duplicate_count} duplicate values, and {empty_count} missing values.')

Series Ticker Symbol has 340 unique values, 0 duplicate values, and 0 missing values.
Series Security has 340 unique values, 0 duplicate values, and 0 missing values.
Series GICS Sector has 11 unique values, 329 duplicate values, and 0 missing values.
Series GICS Sub Industry has 104 unique values, 236 duplicate values, and 0 missing values.
Series Current Price has 336 unique values, 4 duplicate values, and 0 missing values.
Series Price Change has 340 unique values, 0 duplicate values, and 0 missing values.
Series Volatility has 340 unique values, 0 duplicate values, and 0 missing values.
Series ROE has 82 unique values, 258 duplicate values, and 0 missing values.
Series Cash Ratio has 124 unique values, 216 duplicate values, and 0 missing values.
Series Net Cash Flow has 332 unique values, 8 duplicate values, and 0 missing values.
Series Net Income has 335 unique values, 5 duplicate values, and 0 missing values.
Series Earnings Per Share has 268 unique values, 72 duplicate values, and 0 missing values.
Series Estimated Shares Outstanding has 323 unique values, 17 duplicate values, and 0 missing values.
Series P/E Ratio has 295 unique values, 45 duplicate values, and 0 missing values.
Series P/B Ratio has 299 unique values, 41 duplicate values, and 0 missing values.


# Chart high-level assessment for each numerical series
for col in list_numcol:
  mean = df[col].mean()
  median = df[col].median()
  min = df[col].min()
  max = df[col].max()

  this_range = max-min
  print('*****')
  print('For', col, ':')
  print(' * Mean =', mean)
  print(' * Median =' , median)
  print(' * Range =', min, 'to' , max,'(', round(this_range,3), ')')

*****
For Current Price :
 * Mean = 80.86234477216765
 * Median = 59.705
 * Range = 4.5 to 1274.949951 ( 1270.45 )
*****
For Price Change :
 * Mean = 4.078193812467648
 * Median = 4.819504729
 * Range = -47.12969338 to 55.05168339 ( 102.181 )
*****
For Volatility :
 * Mean = 1.5259762538411765
 * Median = 1.3855929255000001
 * Range = 0.733163184 to 4.58004173 ( 3.847 )
*****
For ROE :
 * Mean = 39.59705882352941
 * Median = 15.0
 * Range = 1 to 917 ( 916 )
*****
For Cash Ratio :
 * Mean = 70.02352941176471
 * Median = 47.0
 * Range = 0 to 958 ( 958 )
*****
For Net Cash Flow :
 * Mean = 55537620.5882353
 * Median = 2098000.0
 * Range = -11208000000 to 20764000000 ( 31972000000 )
*****
For Net Income :
 * Mean = 1494384602.9411764
 * Median = 707336000.0
 * Range = -23528000000 to 24442000000 ( 47970000000 )
*****
For Earnings Per Share :
 * Mean = 2.776661764705882
 * Median = 2.895
 * Range = -61.2 to 50.09 ( 111.29 )
*****
For Estimated Shares Outstanding :
 * Mean = 577028337.7540295
 * Median = 309675137.79999995
 * Range = 27672156.86 to 6159292035.0 ( 6131619878.14 )
*****
For P/E Ratio :
 * Mean = 32.612563318264705
 * Median = 20.81987609
 * Range = 2.935450768 to 528.0390742 ( 525.104 )
*****
For P/B Ratio :
 * Mean = -1.7182493633029414
 * Median = -1.0671703205
 * Range = -76.11907749 to 129.0645854 ( 205.184 )


# Check for null values
print(f'EDA dataset has {df_eda.isnull().sum().sum()} null values:',)

# Replace any missing values with NaNs
df_eda = df_eda.replace("?", np.nan)

# Count NaN values
nan_count = df_eda.isnull().sum().sum()

if nan_count > 0:
    print(f"EDA DataFrame has {nan_count} NaN values.")
else:
    print("EDA DataFrame does not have NaN values.")

EDA dataset has 0 null values:
EDA DataFrame does not have NaN values.


# Preview rows with missing data
df_eda[df_eda.isnull().any(axis=1)]


# Preview a random sample of the data
df_eda.sample(n=10, random_state=1)


# Import libraries we need
import matplotlib.pyplot as plt
import seaborn as sns

from sklearn.preprocessing import MinMaxScaler, StandardScaler


# The functions below are defined here so we can avoid repeating similar code for commonly used plots.
# These functions are called throughout this notebook.

# This function prints a barplot
def print_barplot(data, column, width, height, titleText, hue=None):
    '''
    Print a single, sorted barplot with title and labels and optional hue.
    '''

    # Set figure size and title
    plt.figure(figsize=(width, height))
    plt.title(titleText)
    plt.xticks(rotation=90)

    # Sort by values
    data_sorted = data[column].value_counts().sort_values(ascending=False).reset_index()
    data_sorted.columns = ['category', 'counts']

    # Plotting
    if hue is None:
        sns.barplot(x='category', y='counts', data=data_sorted)
    else:
        sns.barplot(x='category', y='counts', hue=hue, data=data_sorted)

    plt.xlabel('Categories')
    plt.ylabel('Counts')
    plt.show()


# This function creates a labeled barplot

def print_labeled_barplot(data, feature, perc=False, n=None):
    '''
    Barplot with percentage at the top

    data: dataframe
    feature: dataframe column
    perc: whether to display percentages instead of count (default is False)
    n: displays the top n category levels (default is None, i.e., display all levels)
    '''

    total = len(data[feature])  # length of the column
    count = data[feature].nunique()
    if n is None:
        plt.figure(figsize=(count + 1, 5))
    else:
        plt.figure(figsize=(n + 1, 5))

    plt.xticks(rotation=90, fontsize=15)
    ax = sns.countplot(
        data=data,
        x=feature,
        palette="Paired",
        order=data[feature].value_counts().index[:n],
    )

    for p in ax.patches:
        if perc == True:
            label = "{:.1f}%".format(
                100 * p.get_height() / total
            )  # percentage of each class of the category
        else:
            label = p.get_height()  # count of each level of the category

        x = p.get_x() + p.get_width() / 2  # width of the plot
        y = p.get_height()  # height of the plot

        ax.annotate(
            label,
            (x, y),
            ha="center",
            va="center",
            size=12,
            xytext=(0, 5),
            textcoords="offset points",
        )  # annotate the percentage

    plt.show()  # show the plot


# Function to plot a boxplot and a histogram on same scale

def histogram_boxplot(data, feature, figsize=(10, 5), kde=False, bins=None):
  '''
  Boxplot and histogram combined

  data: dataframe
  feature: dataframe column
  figsize: size of figure (default (10,5))
  kde: whether to the show density curve (default False)
  bins: number of bins for histogram (default None)
  '''
  # Create a figure with two subplots
  f2, (ax_box2, ax_hist2) = plt.subplots(
      nrows=2,  # Number of rows of the subplot grid
      sharex=True,  # x-axis will be shared among all subplots
      gridspec_kw={'height_ratios': (0.25, 0.75)},  # Ratio of subplot heights
      figsize=figsize,  # Size of the figure
  )

  # Boxplot
  sns.boxplot(
      data=data, x=feature, ax=ax_box2, showmeans=True, color='blue'
  )
  ax_box2.set_title(f'Boxplot of {feature}')  # Title for the boxplot

  # Histogram
  if bins:
      sns.histplot(
          data=data, x=feature, kde=kde, ax=ax_hist2, bins=bins, palette='winter'
      )
  else:
      sns.histplot(
          data=data, x=feature, kde=kde, ax=ax_hist2
      )

  ax_hist2.axvline(
      data[feature].mean(), color='green', linestyle="--", label='Mean'
  )
  ax_hist2.axvline(
      data[feature].median(), color='black', linestyle="-", label='Median'
  )
  ax_hist2.set_title(f'Histogram of {feature}')  # Title for the histogram
  ax_hist2.legend()  # Add legend to the histogram plot

  # Adjust layout
  plt.tight_layout()
  plt.show()


# Function to calculate Z-score
def calculate_z_score(value, mean, std):
    '''Calculate the Z-score for a single value '''
    if std == 0:
        return 0  # Avoid division by zero
    return (value - mean) / std


# Function to print the statistics for each feature
def print_numerical_feature_statistics(data, feature):
    '''
    Show the statistics of a feature.

    data: dataframe
    feature: dataframe column
    '''

    # Calculate Shapiro-Wilks to test normal distribution
    stat, p = shapiro(data[feature])

    # Calculate mean and standard deviation
    mean = data[feature].mean()
    std = data[feature].std()

    # Calculate Z-score for the mean
    z_score = calculate_z_score(mean, mean, std)

    new_stats = {
        'Feature': feature,
        'Mean': mean,
        'Median': data[feature].median(),
        'Min': data[feature].min(),
        'Max': data[feature].max(),
        'Range': data[feature].max() - data[feature].min(),
        'Standard Deviation': std,
        'Variance': data[feature].var(),
        'P-value': p
    }

    new_stats_df = pd.DataFrame([new_stats])

    return new_stats_df


# Analyze critical statistics

# Create empty dataframe to store the vital statistics
stat_columns = [
    'Feature',
    'Mean',
    'Median',
    'Min',
    'Max',
    'Range',
    'Standard Deviation',
    'Variance',
    'P-value']

df_stats = pd.DataFrame(columns=stat_columns)


# Drop ticker symbol and security for this purpose, since they are unique
df_eda.drop(columns=['Ticker Symbol','Security'], axis=1, inplace=True)

# Recreate numerical and categorical column lists
list_catcol = df_eda.select_dtypes(include=['object']).columns.to_list()
list_numcol = df_eda.select_dtypes(exclude=['object']).columns.to_list()


# Populate stats and show them
for feature in list_numcol:
    new_stats_df = print_numerical_feature_statistics(df_eda.select_dtypes(include=['number']), feature)
    df_stats = pd.concat([df_stats, new_stats_df], ignore_index=True)

# Set the display precision
pd.set_option('display.float_format', '{:.6f}'.format)

# Show statistics
df_stats


# Drop numerical columns for charting object histograms
df_eda_plot_cat = df_eda.copy().drop(columns=list_numcol, axis=1)


# Show sectors as a visual

# Create counts and percentages
value_counts = df_eda_plot_cat['GICS Sector'].value_counts().reset_index()
value_counts.columns = ['Category', 'Count']
total_counts = value_counts['Count'].sum()
value_counts['Percentage'] = (value_counts['Count'] / total_counts) * 100
value_counts_sorted = value_counts.sort_values(by='Count', ascending=False)

# Plot sector contributions
plt.figure(figsize=(12, 8))
bar_plot = sns.barplot(x='Category', y='Count', data=value_counts_sorted, order=value_counts_sorted['Category'], hue='Category')

plt.xticks(rotation=90)
plt.xlabel('Categories')
plt.ylabel('Counts')
plt.title('Sector Counts')

for p in bar_plot.patches:
    count = p.get_height()
    percentage = value_counts_sorted["Percentage"].iloc[int(p.get_x() + p.get_width() / 2)]
    bar_plot.annotate(f'{count:.1f}\n({percentage:.1f}%)',
                      (p.get_x() + p.get_width() / 2., count),
                      ha='center', va='bottom')


# Print value contributions for each sector
pd.set_option('display.max_rows', None)
pd.set_option('display.max_columns', None)

display(value_counts_sorted)

# Since our dataset is small, we'll leave this display option intact for later
#pd.set_option('display.max_rows', 10)
#pd.set_option('display.max_columns', 10)


# Show sub sector contributions as a visual

# Create counts and percentages
value_counts_sub = df_eda_plot_cat['GICS Sub Industry'].value_counts().reset_index()
value_counts_sub.columns = ['Category', 'Count']
total_counts_sub = value_counts_sub['Count'].sum()
value_counts_sub['Percentage'] = (value_counts_sub['Count'] / total_counts_sub) * 100
value_counts_sorted_sub = value_counts_sub.sort_values(by='Count', ascending=False)

# Plot sector contributions
plt.figure(figsize=(20, 8))
bar_plot = sns.barplot(x='Category', y='Count', data=value_counts_sorted_sub, order=value_counts_sorted_sub['Category'], hue='Category')

plt.xticks(rotation=90)
plt.xlabel('Categories')
plt.ylabel('Counts')
plt.title('Sector Counts')

plt.show()


# Print value contributions for each sub sector
print('Top 10 Sub Sectors')
display(value_counts_sorted_sub.head(10))

Top 10 Sub Sectors


print('Bottom Sub Sectors')
display(value_counts_sorted_sub[value_counts_sorted_sub['Percentage'] < .3])

Bottom Sub Sectors


# Drop categorical columns this time, so we can chart histograms and boxplots for the numerical features
df_eda_plot_num = df_eda.copy().drop(columns=list_catcol, axis=1)

# Analyze features
for feature in df_eda_plot_num.columns:
  histogram_boxplot(df_eda_plot_num, feature, figsize=(8, 5), kde=True, bins=None)


df_eda.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 340 entries, 0 to 339
Data columns (total 13 columns):
 #   Column                        Non-Null Count  Dtype  
---  ------                        --------------  -----  
 0   GICS Sector                   340 non-null    object 
 1   GICS Sub Industry             340 non-null    object 
 2   Current Price                 340 non-null    float64
 3   Price Change                  340 non-null    float64
 4   Volatility                    340 non-null    float64
 5   ROE                           340 non-null    int64  
 6   Cash Ratio                    340 non-null    int64  
 7   Net Cash Flow                 340 non-null    int64  
 8   Net Income                    340 non-null    int64  
 9   Earnings Per Share            340 non-null    float64
 10  Estimated Shares Outstanding  340 non-null    float64
 11  P/E Ratio                     340 non-null    float64
 12  P/B Ratio                     340 non-null    float64
dtypes: float64(7), int64(4), object(2)
memory usage: 34.7+ KB


# Create new rankings for GICS Sector
unique_sectors = df_eda['GICS Sector'].unique()
ranked_categories = sorted(unique_sectors)
rank_mapping = {}

# Add a dictionary for friendly names
friendly_name_mapping = {}

# Populate mapping and rank
for rank, category in enumerate(ranked_categories):
    rank_mapping[category] = rank + 1
    friendly_name_mapping[category] = f"Rank {rank + 1}: {category}"  # Example friendly name

# Map the rankings and create a new column in the DataFrame
df_eda['GICS Sector Int'] = df_eda['GICS Sector'].map(rank_mapping)


# Drop the non-numerical columns
cols_to_drop = ['GICS Sector', 'GICS Sub Industry']
df_eda_corr = df_eda.copy().drop(columns=cols_to_drop, axis=1)

# Preview new dataframe we will use for correlation heatmap
df_eda_corr


# Plot correlation heatmap
plt.figure(figsize=(15, 7))
sns.heatmap(df_eda_corr.corr(), annot=True, vmin=-1, vmax=1, fmt=".2f", cmap="Spectral")
plt.show()


df_eda.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 340 entries, 0 to 339
Data columns (total 14 columns):
 #   Column                        Non-Null Count  Dtype  
---  ------                        --------------  -----  
 0   GICS Sector                   340 non-null    object 
 1   GICS Sub Industry             340 non-null    object 
 2   Current Price                 340 non-null    float64
 3   Price Change                  340 non-null    float64
 4   Volatility                    340 non-null    float64
 5   ROE                           340 non-null    int64  
 6   Cash Ratio                    340 non-null    int64  
 7   Net Cash Flow                 340 non-null    int64  
 8   Net Income                    340 non-null    int64  
 9   Earnings Per Share            340 non-null    float64
 10  Estimated Shares Outstanding  340 non-null    float64
 11  P/E Ratio                     340 non-null    float64
 12  P/B Ratio                     340 non-null    float64
 13  GICS Sector Int               340 non-null    int64  
dtypes: float64(7), int64(5), object(2)
memory usage: 37.3+ KB


# Normalize the data
categorical_df_eda = df_eda[['GICS Sector', 'GICS Sector Int']] # we want to preserve the integer without scaling
numerical_df_eda = df_eda.drop(columns=['GICS Sector', 'GICS Sub Industry', 'GICS Sector Int'])

# Scale it using min-max method (vs. Z-score), since we want to preserve the relationships between data points
min_max_scaler = MinMaxScaler()
normalized_numerical_data = min_max_scaler.fit_transform(numerical_df_eda)

# Create dataframe for numerical correlation heatmap
normalized_df_eda = pd.DataFrame(normalized_numerical_data, columns=numerical_df_eda.columns)

# Combine back with sector for pairplot that recombines friendly Sector names
final_normalized_df_eda = pd.concat([categorical_df_eda, normalized_df_eda], axis=1)


final_normalized_df_eda.head(5)


#Re-plot the correlation heatmap using scaled data

# Convert scaled Numpy array results to dataframe
normalized_numerical_data_df = pd.DataFrame(normalized_numerical_data, columns=numerical_df_eda.columns)

# Plot new results
plt.figure(figsize=(15, 7))
sns.heatmap(normalized_numerical_data_df.corr(), annot=True, vmin=-1, vmax=1, fmt=".2f", cmap="Spectral")
plt.show()


# Create pairplot, noting the sector values
sns.pairplot(df_eda_corr, diag_kind='kde')

<seaborn.axisgrid.PairGrid at 0x795d44836860>


# Show sector vs. other numeric series using normalized dataframe
for feature in list_numcol:
  plt.figure(figsize=(12, 5))
  sns.boxplot(data=final_normalized_df_eda, x='GICS Sector', y=feature, hue='GICS Sector')
  plt.xticks(rotation=90)
  plt.show()


# Create pivot table for price by sector (counts and averages)
df_eda_sector_pivot = df_eda.pivot_table(
    index='GICS Sector',
    values='Current Price',
    aggfunc=['count', 'mean']  # Calculate both count and mean
)

# Rename the columns for clarity
df_eda_sector_pivot.columns = ['Count', 'Average Current Price']
df_eda_sector_pivot = df_eda_sector_pivot.reset_index()

# Calculate percentages relative to the total count, excluding the total row
total_counts = df_eda_sector_pivot['Count'].sum()
df_eda_sector_pivot['Percentage of Total'] = (df_eda_sector_pivot['Count'] / total_counts) * 100

# Sort the DataFrame by counts in descending order
df_eda_sector_pivot.sort_values(by='Average Current Price', ascending=False, inplace=True)

# Display the combined DataFrame
display(df_eda_sector_pivot)


# Show prior dataframe
df_stats[df_stats['Feature'] == 'Price Change']


# Create pivot table for price change by sector
df_eda_sector_price_change_pivot = df_eda.pivot_table(
    index='GICS Sector',
    values='Price Change',
    aggfunc=['mean', 'min', 'max']
)

df_eda_sector_price_change_pivot['Range'] = df_eda_sector_price_change_pivot['max'] - df_eda_sector_price_change_pivot['min']

df_eda_sector_price_change_pivot.columns = ['Mean Price Change', 'Min', 'Max', 'Range']
df_eda_sector_price_change_pivot.sort_values(by='Mean Price Change', ascending=False, inplace=True)
df_eda_sector_price_change_pivot = df_eda_sector_price_change_pivot.reset_index()

df_eda_sector_price_change_pivot


# Visualize price change
plt.figure(figsize=(10, 5))
sns.barplot(data=df_eda.sort_values(by='Price Change', ascending=False), x='GICS Sector', y='Price Change', hue='GICS Sector')
plt.xticks(rotation=90)
plt.show()


# Visualize volatility
plt.figure(figsize=(10, 5))
sns.barplot(data=df_eda.sort_values(by='Volatility', ascending=False), x='GICS Sector', y='Volatility', hue='GICS Sector')
plt.xticks(rotation=90)
plt.show()


rank_mapping

{'Consumer Discretionary': 1,
 'Consumer Staples': 2,
 'Energy': 3,
 'Financials': 4,
 'Health Care': 5,
 'Industrials': 6,
 'Information Technology': 7,
 'Materials': 8,
 'Real Estate': 9,
 'Telecommunications Services': 10,
 'Utilities': 11}


final_normalized_df_eda


# Plot new results for each sector, using only the top variables and our dictionary from earlier

#normalized_numerical_data_df is the scaled dataframe we created earlier as part of final_normalized_df_eda

cols_to_include = ['GICS Sector Int', 'Net Income', 'Estimated Shares Outstanding',
                   'Earnings Per Share', 'Current Price', 'P/E Ratio',
                   'P/B Ratio', 'Volatility', 'ROE']

# Create an empty DataFrame to store correlations
all_correlations = []

# Plot new results by sector
for friendly_name, rank in rank_mapping.items():
    # Filter the DataFrame for the current sector
    df_temp_corr = final_normalized_df_eda[cols_to_include]
    df_temp_corr = df_temp_corr[df_temp_corr['GICS Sector Int'] == rank]

    # Calculate the correlation matrix
    corr_matrix = df_temp_corr.corr()

    # Store the correlation matrix in a list with sector information
    corr_df = corr_matrix.stack().reset_index()
    corr_df.columns = ['Variable1', 'Variable2', 'Correlation']
    corr_df['Sector'] = friendly_name  # Add the sector name

    # Append to the list
    all_correlations.append(corr_df)

    # Plot the heatmap
    plt.figure(figsize=(15, 7))
    sns.heatmap(corr_matrix, annot=True, vmin=-1, vmax=1, fmt=".2f", cmap="Spectral")
    plt.title(f'{friendly_name}: {rank}')
    plt.show()

# Concatenate all correlation DataFrames into one
all_correlations_df = pd.concat(all_correlations, ignore_index=True)

# Filter for correlations greater than 0.75 (and not equal to 1)
high_corrs = all_correlations_df[(all_correlations_df['Correlation'] > 0.75) & (all_correlations_df['Variable1'] != all_correlations_df['Variable2'])]


# Create pivot table for cash ratio by sector
df_eda_sector_cash_ratio_pivot = df_eda.pivot_table(
    index='GICS Sector',
    values='Cash Ratio',
    aggfunc='mean'
)

# Sort the DataFrame by mean in descending order
df_eda_sector_cash_ratio_pivot.sort_values(by='Cash Ratio', ascending=False, inplace=True)

# Display the combined DataFrame
display(df_eda_sector_cash_ratio_pivot)


# Visualize cash ratio
plt.figure(figsize=(10, 5))
sns.barplot(data=df_eda.sort_values(by='Cash Ratio', ascending=False), x='GICS Sector', y='Cash Ratio', hue='GICS Sector')
plt.xticks(rotation=90)
plt.show()


# Create pivot table for P/E Ratio by sector
df_eda_sector_pe_ratio_pivot = df_eda.pivot_table(
    index='GICS Sector',
    values='P/E Ratio',
    aggfunc='mean'
)

# Sort the DataFrame by mean in descending order
df_eda_sector_pe_ratio_pivot.sort_values(by='P/E Ratio', ascending=False, inplace=True)

# Display the combined DataFrame
display(df_eda_sector_pe_ratio_pivot)


# Visualize P/E ratio
plt.figure(figsize=(10, 5))
sns.barplot(data=df_eda.sort_values(by='P/E Ratio', ascending=False), x='GICS Sector', y='P/E Ratio', hue='GICS Sector')
plt.xticks(rotation=90)
plt.show()


# Import additional libraries
from scipy.spatial.distance import cdist

# Library to compute distances
from scipy.spatial.distance import cdist, pdist

# Library to perform k-means clustering and compute silhouette scores
from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score

# Library for principal component analysis
from sklearn.decomposition import PCA

# Library for scaling; RobustScaler is supposed to work better for datasets with outliers
from sklearn.preprocessing import RobustScaler

# Libraries for visualization
from yellowbrick.cluster import KElbowVisualizer, SilhouetteVisualizer
from sklearn.manifold import TSNE

# Library to perform hierarchical clustering, compute cophenetic correlation, and create dendrograms
from sklearn.cluster import AgglomerativeClustering
from scipy.cluster.hierarchy import dendrogram, linkage, cophenet


# Copy the original dataframe before we begin
df_model = df.copy()


# Check data types
df_model.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 340 entries, 0 to 339
Data columns (total 15 columns):
 #   Column                        Non-Null Count  Dtype  
---  ------                        --------------  -----  
 0   Ticker Symbol                 340 non-null    object 
 1   Security                      340 non-null    object 
 2   GICS Sector                   340 non-null    object 
 3   GICS Sub Industry             340 non-null    object 
 4   Current Price                 340 non-null    float64
 5   Price Change                  340 non-null    float64
 6   Volatility                    340 non-null    float64
 7   ROE                           340 non-null    int64  
 8   Cash Ratio                    340 non-null    int64  
 9   Net Cash Flow                 340 non-null    int64  
 10  Net Income                    340 non-null    int64  
 11  Earnings Per Share            340 non-null    float64
 12  Estimated Shares Outstanding  340 non-null    float64
 13  P/E Ratio                     340 non-null    float64
 14  P/B Ratio                     340 non-null    float64
dtypes: float64(7), int64(4), object(4)
memory usage: 40.0+ KB


# Describe the data. Can also use include=all to also show categorical series.
df_model.describe(include='all').T


# Check each feature to see if we have any single ticker/row as an outlier in all features

features = df_model.columns.difference(['Security'])

# Loop through each feature and print the Ticker Name with the maximum value
for feature in features:
    max_value_row = df_model.loc[df_model[feature].idxmax()]
    min_value_row = df_model.loc[df_model[feature].idxmin()]
    print(f'**{feature}**')
    print(f'  Max:  {max_value_row["Security"]} ({max_value_row[feature]})')
    print(f'  Min:  {min_value_row["Security"]} ({min_value_row[feature]})')

**Cash Ratio**
  Max:  Facebook (958)
  Min:  Charter Communications (0)
**Current Price**
  Max:  Priceline.com Inc (1274.949951)
  Min:  Chesapeake Energy (4.5)
**Earnings Per Share**
  Max:  Priceline.com Inc (50.09)
  Min:  Apache Corporation (-61.2)
**Estimated Shares Outstanding**
  Max:  Pfizer Inc. (6159292035.0)
  Min:  Mettler Toledo (27672156.86)
**GICS Sector**
  Max:  Ameren Corp (Utilities)
  Min:  Amazon.com Inc (Consumer Discretionary)
**GICS Sub Industry**
  Max:  American Water Works Company Inc (Water Utilities)
  Min:  Interpublic Group (Advertising)
**Net Cash Flow**
  Max:  Bank of America Corp (20764000000)
  Min:  Citigroup Inc. (-11208000000)
**Net Income**
  Max:  JPMorgan Chase & Co. (24442000000)
  Min:  Apache Corporation (-23528000000)
**P/B Ratio**
  Max:  Alliance Data Systems (129.0645854)
  Min:  Charter Communications (-76.11907749)
**P/E Ratio**
  Max:  Amazon.com Inc (528.0390742)
  Min:  United Continental Holdings (2.935450768)
**Price Change**
  Max:  First Solar Inc (55.05168339)
  Min:  Kinder Morgan (-47.12969338)
**ROE**
  Max:  Apache Corporation (917)
  Min:  Concho Resources (1)
**Ticker Symbol**
  Max:  Zoetis (ZTS)
  Min:  American Airlines Group (AAL)
**Volatility**
  Max:  Southwestern Energy (4.58004173)
  Min:  McDonald's Corp. (0.733163184)


# Create tight boxplots to show outliers
plt.figure(figsize=(12, 10))

num_cols = df_model.select_dtypes(include=np.number).columns.tolist()

for i, col in enumerate(num_cols):
    plt.subplot(3, 4, i + 1)
    plt.boxplot(df_model[col], whis=1.5)
    plt.tight_layout()
    plt.title(col)

plt.show()


df_model.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 340 entries, 0 to 339
Data columns (total 15 columns):
 #   Column                        Non-Null Count  Dtype  
---  ------                        --------------  -----  
 0   Ticker Symbol                 340 non-null    object 
 1   Security                      340 non-null    object 
 2   GICS Sector                   340 non-null    object 
 3   GICS Sub Industry             340 non-null    object 
 4   Current Price                 340 non-null    float64
 5   Price Change                  340 non-null    float64
 6   Volatility                    340 non-null    float64
 7   ROE                           340 non-null    int64  
 8   Cash Ratio                    340 non-null    int64  
 9   Net Cash Flow                 340 non-null    int64  
 10  Net Income                    340 non-null    int64  
 11  Earnings Per Share            340 non-null    float64
 12  Estimated Shares Outstanding  340 non-null    float64
 13  P/E Ratio                     340 non-null    float64
 14  P/B Ratio                     340 non-null    float64
dtypes: float64(7), int64(4), object(4)
memory usage: 40.0+ KB


# One-hot encode variables for Sector and Sub Sector: Determined we don't want to do this
#df_model_clean = pd.get_dummies(df_model_clean, columns=['GICS Sector'])
#df_model_clean = pd.get_dummies(df_model_clean, columns=['GICS Sub Industry'])


# Drop unique values: Determined we don't want to do this
#df_model_clean = df_model_clean.drop(['Ticker Symbol', 'Security'], axis=1)


df_model.describe(include='all').T


# Initialize the RobustScaler
scaler = RobustScaler()

subset = df_model.select_dtypes(include=['number']).copy()

# Scale the data
subset_scaled = scaler.fit_transform(subset)

# Convert to DataFrame
df_model_scaled = pd.DataFrame(subset_scaled, columns=subset.columns)


# Measure execution time
%%time

# Identify desired cluster count
df_k_means = df_model_scaled.copy()

clusters = range(1, 16)
meanDistortions = []

# Loop through cluster values
for k in clusters:
    model_k_means = KMeans(n_clusters=k)
    model_k_means.fit(df_k_means)
    prediction = model_k_means.predict(df_k_means)
    distortion = (
        sum(
            np.min(cdist(df_k_means, model_k_means.cluster_centers_, 'euclidean'), axis=1)
        )
        / df_k_means.shape[0]
    )

    meanDistortions.append(distortion)

    print('# of Clusters:', k, '\tAverage Distortion:', distortion)

plt.plot(clusters, meanDistortions, 'bx-')
plt.xlabel('k')
plt.ylabel('Avg Distortion')
plt.title('Selecting k with Elbow Method')
plt.show()

# of Clusters: 1 	Average Distortion: 6.165015784179755
# of Clusters: 2 	Average Distortion: 5.519913922135517
# of Clusters: 3 	Average Distortion: 5.301175199731372
# of Clusters: 4 	Average Distortion: 5.128821925696185
# of Clusters: 5 	Average Distortion: 4.852367478608126
# of Clusters: 6 	Average Distortion: 4.564025418031926
# of Clusters: 7 	Average Distortion: 4.454884657269399
# of Clusters: 8 	Average Distortion: 4.221774350762034
# of Clusters: 9 	Average Distortion: 4.191296800846647
# of Clusters: 10 	Average Distortion: 4.079244586736443
# of Clusters: 11 	Average Distortion: 3.829791083252664
# of Clusters: 12 	Average Distortion: 3.753949415523762
# of Clusters: 13 	Average Distortion: 3.6369603635468146
# of Clusters: 14 	Average Distortion: 3.596860202066002
# of Clusters: 15 	Average Distortion: 3.445799561982341

CPU times: user 409 ms, sys: 112 ms, total: 521 ms
Wall time: 488 ms


# Fit the model to the visualizer
model_k_means_2 = KMeans(random_state=1)
visualizer = KElbowVisualizer(model_k_means_2, k=(1, 16), timings=True)
visualizer.fit(df_k_means)
visualizer.show()

<Axes: title={'center': 'Distortion Score Elbow for KMeans Clustering'}, xlabel='k', ylabel='distortion score'>


# Check silhouette scores
sil_score = []

cluster_list = range(2, 16)

for n_clusters in cluster_list:
    clusterer = KMeans(n_clusters=n_clusters, random_state=1)
    preds = clusterer.fit_predict((df_k_means))
    score = silhouette_score(df_k_means, preds)
    sil_score.append(score)
    print('n_clusters={}: Silhouette score={})'.format(n_clusters, score))

plt.plot(cluster_list, sil_score)
plt.show()

n_clusters=2: Silhouette score=0.7757324276239033)
n_clusters=3: Silhouette score=0.7454764840420494)
n_clusters=4: Silhouette score=0.4621955535259713)
n_clusters=5: Silhouette score=0.4829025199901958)
n_clusters=6: Silhouette score=0.4813819928027789)
n_clusters=7: Silhouette score=0.49576783422468745)
n_clusters=8: Silhouette score=0.411864134740954)
n_clusters=9: Silhouette score=0.4012633404812339)
n_clusters=10: Silhouette score=0.3763626634759577)
n_clusters=11: Silhouette score=0.3505016857108297)
n_clusters=12: Silhouette score=0.3613349297613386)
n_clusters=13: Silhouette score=0.3806084017704855)
n_clusters=14: Silhouette score=0.3183846743555661)
n_clusters=15: Silhouette score=0.27805939414564607)


# Visualize the silhouette score for various values between 5 and 8

for i in range(5, 9):
    visualizer = SilhouetteVisualizer(KMeans(i, random_state=1))
    visualizer.fit(df_k_means)
    visualizer.show()


# Run again for the ideal cluster count for k-means
kmeans = KMeans(n_clusters=5, random_state=1)
kmeans.fit(df_model_scaled)

KMeans(n_clusters=5, random_state=1)

KMeans(n_clusters=5, random_state=1)


# Apply the k-means to a final model
df_kmean = df_model.copy()

# Add labels
df_kmean['Cluster'] = kmeans.labels_
df_model_scaled['Cluster'] = kmeans.labels_


# Show counts by cluster
display(df_kmean['Cluster'].value_counts())


# Show the securities in each cluster
for cluster in df_kmean['Cluster'].unique():
    cluster_securities = df_kmean[df_kmean['Cluster'] == cluster]['Security']
    print(f'Cluster {cluster}: {", ".join(cluster_securities)}')

Cluster 1: American Airlines Group, AbbVie, Devon Energy Corp., Freeport-McMoran Cp & Gld, IDEXX Laboratories, Masco Corp., Moody's Corp, Altria Group Inc, Newfield Exploration Co, Pitney-Bowes, Sherwin-Williams, Southwestern Energy, United Parcel Service, Wynn Resorts Ltd, Cimarex Energy, Yum! Brands Inc
Cluster 3: Abbott Laboratories, Adobe Systems Inc, Analog Devices, Inc., Archer-Daniels-Midland Co, Alliance Data Systems, Ameren Corp, American Electric Power, AFLAC Inc, American International Group, Inc., Apartment Investment & Mgmt, Assurant Inc, Arthur J. Gallagher & Co., Akamai Technologies Inc, Albemarle Corp, Alaska Air Group Inc, Allstate Corp, Alexion Pharmaceuticals, Applied Materials Inc, AMETEK Inc, Affiliated Managers Group Inc, Amgen Inc, Ameriprise Financial, American Tower Corp A, Amazon.com Inc, AutoNation Inc, Anthem Inc., Aon plc, Amphenol Corp, Arconic Inc, Activision Blizzard, AvalonBay Communities, Inc., Broadcom, American Water Works Company Inc, American Express Co, Boeing Company, Baxter International Inc., BB&T Corporation, Bard (C.R.) Inc., Baker Hughes Inc, BIOGEN IDEC Inc., The Bank of New York Mellon Corp., Ball Corp, Boston Scientific, BorgWarner, Boston Properties, Caterpillar Inc., Chubb Limited, CBRE Group, Crown Castle International Corp., Carnival Corp., Celgene Corp., CF Industries Holdings Inc, Citizens Financial Group, Church & Dwight, C. H. Robinson Worldwide, CIGNA Corp., Cincinnati Financial, Comerica Inc., CME Group Inc., Chipotle Mexican Grill, Cummins Inc., CMS Energy, Centene Corporation, CenterPoint Energy, Capital One Financial, Cabot Oil & Gas, The Cooper Companies, CSX Corp., CenturyLink Inc, Cognizant Technology Solutions, Citrix Systems, CVS Health, Chevron Corp., Concho Resources, Dominion Resources, Delta Air Lines, Du Pont (E.I.), Deere & Co., Discover Financial Services, Quest Diagnostics, Danaher Corp., The Walt Disney Company, Discovery Communications-A, Discovery Communications-C, Delphi Automotive, Digital Realty Trust, Dun & Bradstreet, Dover Corp., Dr Pepper Snapple Group, Duke Energy, DaVita Inc., Ecolab Inc., Consolidated Edison, Equifax Inc., Edison Int'l, Eastman Chemical, EOG Resources, Equinix, Equity Residential, EQT Corporation, Eversource Energy, Essex Property Trust, Inc., E*Trade, Eaton Corporation, Entergy Corp., Edwards Lifesciences, Exelon Corp., Expeditors Int'l, Expedia Inc., Extra Space Storage, Ford Motor, Fastenal Co, Facebook, Fortune Brands Home & Security, FirstEnergy Corp, Fidelity National Information Services, Fiserv Inc, FLIR Systems, Fluor Corp., Flowserve Corporation, FMC Corporation, Federal Realty Investment Trust, First Solar Inc, Frontier Communications, General Dynamics, General Growth Properties Inc., Gilead Sciences, Corning Inc., Genuine Parts, Garmin Ltd., Goodyear Tire & Rubber, Grainger (W.W.) Inc., Halliburton Co., Hasbro Inc., Huntington Bancshares, HCA Holdings, Welltower Inc., HCP Inc., Hess Corporation, Hartford Financial Svc.Gp., Harley-Davidson, Honeywell Int'l Inc., Hewlett Packard Enterprise, HP Inc., Hormel Foods Corp., Henry Schein, Host Hotels & Resorts, The Hershey Company, Humana Inc., International Business Machines, Intl Flavors & Fragrances, International Paper, Interpublic Group, Iron Mountain Incorporated, Intuitive Surgical Inc., Illinois Tool Works, Invesco Ltd., J. B. Hunt Transport Services, Jacobs Engineering Group, Juniper Networks, Kimco Realty, Kinder Morgan, Kansas City Southern, Leggett & Platt, Lennar Corp., Laboratory Corp. of America Holding, LKQ Corporation, L-3 Communications Holdings, Lilly (Eli) & Co., Lockheed Martin Corp., Alliant Energy Corp, Leucadia National Corp., Southwest Airlines, Level 3 Communications, LyondellBasell, Mastercard Inc., Mid-America Apartments, Macerich, Marriott Int'l., Mattel Inc., McDonald's Corp., Mondelez International, MetLife Inc., Mohawk Industries, Mead Johnson, McCormick & Co., Martin Marietta Materials, Marsh & McLennan, 3M Company, Monster Beverage, The Mosaic Company, Marathon Petroleum, Merck & Co., Marathon Oil Corp., M&T Bank Corp., Mettler Toledo, Murphy Oil, Mylan N.V., Navient, Noble Energy Inc, NASDAQ OMX Group, NextEra Energy, Newmont Mining Corp. (Hldg. Co.), Netflix Inc., Nielsen Holdings, National Oilwell Varco Inc., Norfolk Southern Corp., Northern Trust Corp., Nucor Corp., Newell Brands, Realty Income Corporation, ONEOK, Omnicom Group, O'Reilly Automotive, Occidental Petroleum, People's United Financial, PACCAR Inc., PG&E Corp., Priceline.com Inc, Public Serv. Enterprise Inc., PepsiCo Inc., Principal Financial Group, Procter & Gamble, Progressive Corp., Pulte Homes Inc., Philip Morris International, PNC Financial Services, Pentair Ltd., Pinnacle West Capital, PPG Industries, PPL Corp., Prudential Financial, Phillips 66, Quanta Services Inc., Praxair Inc., PayPal, Ryder System, Royal Caribbean Cruises Ltd, Regeneron, Robert Half International, Roper Industries, Range Resources Corp., Republic Services Inc, SCANA Corp, Charles Schwab Corporation, Spectra Energy Corp., Sealed Air, SL Green Realty, Scripps Networks Interactive Inc., Southern Co., Simon Property Group Inc, Stericycle Inc, Sempra Energy, SunTrust Banks, State Street Corp., Skyworks Solutions, Synchrony Financial, Stryker Corp., Molson Coors Brewing Company, Teradata Corp., Tegna, Inc., Torchmark Corp., Thermo Fisher Scientific, TripAdvisor, The Travelers Companies Inc., Tractor Supply Company, Tyson Foods, Tesoro Petroleum Co., Total System Services, Texas Instruments, Under Armour, United Continental Holdings, UDR Inc, Universal Health Services, Inc., United Health Group Inc., Unum Group, Union Pacific, United Technologies, Varian Medical Systems, Valero Energy, Vulcan Materials, Vornado Realty Trust, Verisk Analytics, Verisign Inc., Vertex Pharmaceuticals Inc, Ventas Inc, Waters Corporation, Wec Energy Group Inc, Whirlpool Corp., Waste Management Inc., Williams Cos., Western Union Co, Weyerhaeuser Corp., Wyndham Worldwide, Xcel Energy Inc, XL Capital, Dentsply Sirona, Xerox Corp., Xylem Inc., Yahoo Inc., Zimmer Biomet Holdings, Zions Bancorp, Zoetis
Cluster 2: Allegion, Apache Corporation, Chesapeake Energy, Charter Communications, Colgate-Palmolive, Kimberly-Clark, S&P Global, Inc.
Cluster 0: Anadarko Petroleum Corp, Bristol-Myers Squibb, Citigroup Inc., eBay Inc., General Motors, JPMorgan Chase & Co., Coca Cola Company, Pfizer Inc., AT&T Inc, Verizon Communications, Wells Fargo, Exxon Mobil Corp.
Cluster 4: Bank of America Corp, Intel Corp.


# Create a crosstab for sectors and clusters
df_kmean_crosstab_with_totals = pd.crosstab(df_kmean['Cluster'], df_kmean['GICS Sector'], margins=True, margins_name="Total")

# Display the crosstab with totals
df_kmean_crosstab_with_totals.T  # Transpose to switch rows/columns if needed


# Show means for our original columns by Cluster
#df_model_k_means_grouped = df_model_k_means.groupby('Cluster').mean()

# Loop through the selected columns and plot boxplots
for col in list_numcol:
    if col in df_kmean.columns:  # Ensure the column exists in the DataFrame
        plt.figure(figsize=(8, 5))
        sns.boxplot(data=df_kmean, x='Cluster', y=col, hue='Cluster', palette='viridis', legend='full')
        plt.xticks(rotation=90)
        plt.title(f'Boxplot of {col} by Cluster')
        plt.show()


df_hierarchical = df_model_scaled.copy()


#Measure execution time
%%time

# Create list of distance metrics
distance_metrics = ['euclidean', 'chebyshev', 'mahalanobis', 'cityblock']

# Create list of linkage methods
linkage_methods = ['single', 'complete', 'average', 'centroid', 'ward', 'weighted']

high_cophenet_corr = 0
high_dm_lm = [0, 0]

# Calculate the cophentics for each combination of distance type and linkage method
for dm in distance_metrics:
    for lm in linkage_methods:
        try:
            # Try to compute linkage and cophenet correlation
            Z = linkage(df_hierarchical, metric=dm, method=lm)
            c, coph_dists = cophenet(Z, pdist(df_hierarchical))

            # Print cophenetic correlation result
            print(
                'Cophenetic correlation for {} distance and {} linkage is {}.'.format(
                    dm.capitalize(), lm, c
                )
            )

            # Update the highest cophenetic correlation
            if high_cophenet_corr < c:
                high_cophenet_corr = c
                high_dm_lm[0] = dm
                high_dm_lm[1] = lm

        except Exception as e:
            # Not all distance types can be used with all linkage types, so we need to gracefully handle errors
            print(f'Error for {dm.capitalize()} distance and {lm} linkage: {e}')

Cophenetic correlation for Euclidean distance and single linkage is 0.9515911691498813.
Cophenetic correlation for Euclidean distance and complete linkage is 0.8973300286572553.
Cophenetic correlation for Euclidean distance and average linkage is 0.9456134806319734.
Cophenetic correlation for Euclidean distance and centroid linkage is 0.9468040178955592.
Cophenetic correlation for Euclidean distance and ward linkage is 0.8184834889632843.
Cophenetic correlation for Euclidean distance and weighted linkage is 0.8824190968054058.
Cophenetic correlation for Chebyshev distance and single linkage is 0.944359706184292.
Cophenetic correlation for Chebyshev distance and complete linkage is 0.8863927240886589.
Cophenetic correlation for Chebyshev distance and average linkage is 0.9381981458216698.
Error for Chebyshev distance and centroid linkage: `method=centroid` requires the distance metric to be Euclidean
Error for Chebyshev distance and ward linkage: `method=ward` requires the distance metric to be Euclidean
Cophenetic correlation for Chebyshev distance and weighted linkage is 0.8839528285551623.
Cophenetic correlation for Mahalanobis distance and single linkage is 0.8688554364012083.
Cophenetic correlation for Mahalanobis distance and complete linkage is 0.49533463354436585.
Cophenetic correlation for Mahalanobis distance and average linkage is 0.8380995772123909.
Error for Mahalanobis distance and centroid linkage: `method=centroid` requires the distance metric to be Euclidean
Error for Mahalanobis distance and ward linkage: `method=ward` requires the distance metric to be Euclidean
Cophenetic correlation for Mahalanobis distance and weighted linkage is 0.7167075784099872.
Cophenetic correlation for Cityblock distance and single linkage is 0.9209089428269329.
Cophenetic correlation for Cityblock distance and complete linkage is 0.8440926567175726.
Cophenetic correlation for Cityblock distance and average linkage is 0.9363186631034428.
Error for Cityblock distance and centroid linkage: `method=centroid` requires the distance metric to be Euclidean
Error for Cityblock distance and ward linkage: `method=ward` requires the distance metric to be Euclidean
Cophenetic correlation for Cityblock distance and weighted linkage is 0.8593697074578455.
CPU times: user 197 ms, sys: 102 ms, total: 299 ms
Wall time: 185 ms


# Print the ideal combination
print(
    'Highest cophenetic correlation is {}, which is obtained with {} distance and {} linkage.'.format(
        high_cophenet_corr, high_dm_lm[0].capitalize(), high_dm_lm[1]
    )
)

Highest cophenetic correlation is 0.9515911691498813, which is obtained with Euclidean distance and single linkage.


# View dendograms

# Create list of linkage methods
linkage_methods = ['single', 'complete', 'average', 'centroid', 'ward', 'weighted'] ## Complete the code to add linkages

# Create lists to save results
compare_cols = ['Linkage', 'Cophenetic Coefficient']

# Create subplot
fig, axs = plt.subplots(len(linkage_methods), 1, figsize=(15, 30))

# Loop through thelinkage methods and plot dendrogram and calculate cophenetic correlation
for i, method in enumerate(linkage_methods):
    Z = linkage(df_hierarchical, metric='euclidean', method=method)

    dendrogram(Z, ax=axs[i])
    axs[i].set_title(f'Dendrogram ({method.capitalize()} Linkage)')

    coph_corr, coph_dist = cophenet(Z, pdist(df_hierarchical))
    axs[i].annotate(
        f'Cophenetic\nCorrelation\n{coph_corr:0.2f}',
        (0.80, 0.80),
        xycoords='axes fraction',
    )


# Fit the model with the desired number of clusters and selected distance and linkage type
# Note that affinity is now called metric in the function below
df_model_hc_5 = AgglomerativeClustering(n_clusters=5, metric='euclidean', linkage='complete')
df_model_hc_5.fit(df_hierarchical)

AgglomerativeClustering(linkage='complete', n_clusters=5)

AgglomerativeClustering(linkage='complete', n_clusters=5)


# Apply the hierarchical clustering to a final model
df_hc = df_model.copy()

# Add labels
df_hierarchical['Cluster'] = df_model_hc_5.labels_
df_hc['Cluster'] = df_model_hc_5.labels_


# Group and count the results
display(df_hc['Cluster'].value_counts())


# Show the securities in each cluster
for cluster in df_hc['Cluster'].unique():
    cluster_securities = df_hc[df_hc['Cluster'] == cluster]['Security']
    print(f'Cluster {cluster}: {", ".join(cluster_securities)}')

Cluster 1: American Airlines Group, AbbVie, Abbott Laboratories, Adobe Systems Inc, Analog Devices, Inc., Archer-Daniels-Midland Co, Alliance Data Systems, Ameren Corp, American Electric Power, AFLAC Inc, American International Group, Inc., Apartment Investment & Mgmt, Assurant Inc, Arthur J. Gallagher & Co., Akamai Technologies Inc, Albemarle Corp, Alaska Air Group Inc, Allstate Corp, Applied Materials Inc, AMETEK Inc, Affiliated Managers Group Inc, Amgen Inc, Ameriprise Financial, American Tower Corp A, AutoNation Inc, Anthem Inc., Aon plc, Anadarko Petroleum Corp, Amphenol Corp, Arconic Inc, Activision Blizzard, AvalonBay Communities, Inc., Broadcom, American Water Works Company Inc, American Express Co, Boeing Company, Baxter International Inc., BB&T Corporation, Bard (C.R.) Inc., Baker Hughes Inc, BIOGEN IDEC Inc., The Bank of New York Mellon Corp., Ball Corp, Bristol-Myers Squibb, Boston Scientific, BorgWarner, Boston Properties, Caterpillar Inc., Chubb Limited, CBRE Group, Crown Castle International Corp., Carnival Corp., Celgene Corp., CF Industries Holdings Inc, Citizens Financial Group, Church & Dwight, C. H. Robinson Worldwide, CIGNA Corp., Cincinnati Financial, Comerica Inc., CME Group Inc., Chipotle Mexican Grill, Cummins Inc., CMS Energy, Centene Corporation, CenterPoint Energy, Capital One Financial, Cabot Oil & Gas, The Cooper Companies, CSX Corp., CenturyLink Inc, Cognizant Technology Solutions, Citrix Systems, CVS Health, Chevron Corp., Concho Resources, Dominion Resources, Delta Air Lines, Du Pont (E.I.), Deere & Co., Discover Financial Services, Quest Diagnostics, Danaher Corp., The Walt Disney Company, Discovery Communications-A, Discovery Communications-C, Delphi Automotive, Digital Realty Trust, Dun & Bradstreet, Dover Corp., Dr Pepper Snapple Group, Duke Energy, DaVita Inc., Devon Energy Corp., eBay Inc., Ecolab Inc., Consolidated Edison, Equifax Inc., Edison Int'l, Eastman Chemical, EOG Resources, Equinix, Equity Residential, EQT Corporation, Eversource Energy, Essex Property Trust, Inc., E*Trade, Eaton Corporation, Entergy Corp., Edwards Lifesciences, Expeditors Int'l, Expedia Inc., Extra Space Storage, Fastenal Co, Facebook, Fortune Brands Home & Security, Freeport-McMoran Cp & Gld, FirstEnergy Corp, Fidelity National Information Services, Fiserv Inc, FLIR Systems, Fluor Corp., Flowserve Corporation, FMC Corporation, Federal Realty Investment Trust, First Solar Inc, Frontier Communications, General Dynamics, General Growth Properties Inc., Gilead Sciences, Corning Inc., General Motors, Genuine Parts, Garmin Ltd., Goodyear Tire & Rubber, Grainger (W.W.) Inc., Hasbro Inc., Huntington Bancshares, HCA Holdings, Welltower Inc., HCP Inc., Hess Corporation, Hartford Financial Svc.Gp., Harley-Davidson, Honeywell Int'l Inc., HP Inc., Hormel Foods Corp., Henry Schein, Host Hotels & Resorts, The Hershey Company, Humana Inc., International Business Machines, IDEXX Laboratories, Intl Flavors & Fragrances, International Paper, Interpublic Group, Iron Mountain Incorporated, Intuitive Surgical Inc., Illinois Tool Works, Invesco Ltd., J. B. Hunt Transport Services, Jacobs Engineering Group, Juniper Networks, Kimco Realty, Kinder Morgan, Coca Cola Company, Kansas City Southern, Leggett & Platt, Lennar Corp., Laboratory Corp. of America Holding, LKQ Corporation, L-3 Communications Holdings, Lilly (Eli) & Co., Lockheed Martin Corp., Alliant Energy Corp, Leucadia National Corp., Southwest Airlines, Level 3 Communications, LyondellBasell, Mastercard Inc., Mid-America Apartments, Macerich, Marriott Int'l., Masco Corp., Mattel Inc., Moody's Corp, Mondelez International, MetLife Inc., Mohawk Industries, Mead Johnson, McCormick & Co., Martin Marietta Materials, Marsh & McLennan, 3M Company, Monster Beverage, Altria Group Inc, The Mosaic Company, Marathon Petroleum, Merck & Co., Marathon Oil Corp., M&T Bank Corp., Mettler Toledo, Murphy Oil, Mylan N.V., Navient, Noble Energy Inc, NASDAQ OMX Group, NextEra Energy, Newmont Mining Corp. (Hldg. Co.), Newfield Exploration Co, Nielsen Holdings, National Oilwell Varco Inc., Norfolk Southern Corp., Nucor Corp., Newell Brands, Realty Income Corporation, ONEOK, Omnicom Group, O'Reilly Automotive, Occidental Petroleum, People's United Financial, Pitney-Bowes, PACCAR Inc., PG&E Corp., Public Serv. Enterprise Inc., PepsiCo Inc., Pfizer Inc., Principal Financial Group, Procter & Gamble, Progressive Corp., Pulte Homes Inc., Philip Morris International, PNC Financial Services, Pentair Ltd., Pinnacle West Capital, PPG Industries, PPL Corp., Prudential Financial, Phillips 66, Quanta Services Inc., Praxair Inc., PayPal, Ryder System, Royal Caribbean Cruises Ltd, Regeneron, Robert Half International, Roper Industries, Range Resources Corp., Republic Services Inc, SCANA Corp, Charles Schwab Corporation, Spectra Energy Corp., Sealed Air, Sherwin-Williams, SL Green Realty, Scripps Networks Interactive Inc., Southern Co., Simon Property Group Inc, Stericycle Inc, Sempra Energy, SunTrust Banks, State Street Corp., Skyworks Solutions, Southwestern Energy, Synchrony Financial, Stryker Corp., AT&T Inc, Molson Coors Brewing Company, Teradata Corp., Tegna, Inc., Torchmark Corp., Thermo Fisher Scientific, TripAdvisor, The Travelers Companies Inc., Tractor Supply Company, Tyson Foods, Tesoro Petroleum Co., Total System Services, Texas Instruments, Under Armour, United Continental Holdings, UDR Inc, Universal Health Services, Inc., United Health Group Inc., Unum Group, Union Pacific, United Parcel Service, United Technologies, Varian Medical Systems, Valero Energy, Vulcan Materials, Vornado Realty Trust, Verisk Analytics, Verisign Inc., Vertex Pharmaceuticals Inc, Ventas Inc, Waters Corporation, Wec Energy Group Inc, Wells Fargo, Whirlpool Corp., Waste Management Inc., Williams Cos., Western Union Co, Weyerhaeuser Corp., Wyndham Worldwide, Wynn Resorts Ltd, Cimarex Energy, Xcel Energy Inc, XL Capital, Exxon Mobil Corp., Dentsply Sirona, Xerox Corp., Xylem Inc., Yahoo Inc., Yum! Brands Inc, Zimmer Biomet Holdings, Zions Bancorp, Zoetis
Cluster 2: Allegion, Apache Corporation, Chesapeake Energy, Charter Communications, Colgate-Palmolive, Kimberly-Clark, S&P Global, Inc.
Cluster 0: Alexion Pharmaceuticals, Amazon.com Inc, Exelon Corp., Ford Motor, Halliburton Co., Hewlett Packard Enterprise, McDonald's Corp., Netflix Inc., Northern Trust Corp., Priceline.com Inc
Cluster 4: Bank of America Corp, Intel Corp.
Cluster 3: Citigroup Inc., JPMorgan Chase & Co., Verizon Communications


# Loop through the selected columns and plot boxplots
for col in list_numcol:
    if col in df_hc.columns:  # Ensure the column exists in the DataFrame
        plt.figure(figsize=(8, 5))
        sns.boxplot(data=df_hc, x='Cluster', y=col, hue='Cluster', palette='viridis', legend='full')
        plt.xticks(rotation=90)
        plt.title(f"Boxplot of {col} by Cluster")
        plt.show()


# Copy datafrase
df_pca = df_model_scaled.copy()


# Reduce to higher dimensions first with PCA, using n_components from 2 to 10, and kmeans results *cluster=5)
for i in np.arange(2, 11):

  # Initialize PCA model
  pca = PCA(n_components=i)  # or any number you want
  X_pca = pca.fit_transform(df_pca)

  # Then apply t-SNE
  tsne = TSNE(n_components=2, random_state=8)
  X_reduced = tsne.fit_transform(X_pca)

  # Perform clustering on the reduced data (using KMeans cluster count here)
  kmeans = KMeans(n_clusters=5, random_state=8)
  clusters = kmeans.fit_predict(X_reduced)

  # Put t-SNE results in a dataframe
  df_tsne = pd.DataFrame(data=X_reduced, columns=['Component 1', 'Component 2'])

  # Add the cluster labels to the dataframe
  df_tsne['Cluster'] = clusters

  # Show it in a scatterplot
  plt.figure(figsize=(10, 8))
  sns.scatterplot(data=df_tsne, x='Component 1', y='Component 2',  hue='Cluster', palette='viridis', legend='full')

  # Customize the plot
  plt.title(f't-SNE Visualization of K-Means Clusters: {i}')
  plt.xlabel('Component 1')
  plt.ylabel('Component 2')
  plt.legend(title='Cluster')
  plt.show()


# Reduce to higher dimensions first with PCA, using n_components from 2 to 10
for i in np.arange(2, 11):

  # Initialize PCA model
  pca = PCA(n_components=i)
  X_pca = pca.fit_transform(df_pca)

  # Then apply t-SNE
  tsne = TSNE(n_components=2, random_state=8)
  X_reduced = tsne.fit_transform(X_pca)

  # Perform hierarchical clustering (AgglomerativeClustering) on the reduced data
  hierarchical = AgglomerativeClustering(n_clusters=5)  # Set number of clusters to 5
  clusters = hierarchical.fit_predict(X_reduced)

  # Put t-SNE results in a dataframe
  df_tsne = pd.DataFrame(data=X_reduced, columns=['Component 1', 'Component 2'])

  # Add the cluster labels to the dataframe
  df_tsne['Cluster'] = clusters

  # Show it in a scatterplot
  plt.figure(figsize=(10, 8))
  sns.scatterplot(data=df_tsne, x='Component 1', y='Component 2',  hue='Cluster', palette='viridis', legend='full')

  # Customize the plot
  plt.title(f't-SNE Visualization of Hierarchical Clustering (n_clusters=5): {i}')
  plt.xlabel('Component 1')
  plt.ylabel('Component 2')
  plt.legend(title='Cluster')
  plt.show()


# Create cluster-specific dataframes
df_kmean_cluster_0 = df_kmean[df_kmean['Cluster'] == 0]
df_kmean_cluster_1 = df_kmean[df_kmean['Cluster'] == 1]
df_kmean_cluster_2 = df_kmean[df_kmean['Cluster'] == 2]
df_kmean_cluster_3 = df_kmean[df_kmean['Cluster'] == 3]
df_kmean_cluster_4 = df_kmean[df_kmean['Cluster'] == 4]

list_clusters = [0,1,2,3,4]


# View details for Cluster means
for col in num_cols:
  print(f'\n****** Feature {col} ******')
  for cluster in list_clusters:
    cluster_data = df_kmean[df_kmean['Cluster'] == cluster]
    print(f'Cluster {cluster} mean for {col}: {cluster_data[col].mean()}')

****** Feature Current Price ******
Cluster 0 mean for Current Price: 48.73499975000001
Cluster 1 mean for Current Price: 64.210386025625
Cluster 2 mean for Current Price: 84.35571614285713
Cluster 3 mean for Current Price: 83.29782519843894
Cluster 4 mean for Current Price: 25.6400005

****** Feature Price Change ******
Cluster 0 mean for Price Change: 5.318747360916667
Cluster 1 mean for Price Change: -2.1037755721874998
Cluster 2 mean for Price Change: 3.854980541142857
Cluster 3 mean for Price Change: 4.313401507676567
Cluster 4 mean for Price Change: 11.2379082245

****** Feature Volatility ******
Cluster 0 mean for Volatility: 1.2708920399166668
Cluster 1 mean for Volatility: 2.11978607225
Cluster 2 mean for Volatility: 1.827670343428571
Cluster 3 mean for Volatility: 1.499096510122112
Cluster 4 mean for Volatility: 1.32235485

****** Feature ROE ******
Cluster 0 mean for ROE: 26.0
Cluster 1 mean for ROE: 178.5
Cluster 2 mean for ROE: 633.5714285714286
Cluster 3 mean for ROE: 19.257425742574256
Cluster 4 mean for ROE: 12.5

****** Feature Cash Ratio ******
Cluster 0 mean for Cash Ratio: 71.75
Cluster 1 mean for Cash Ratio: 64.125
Cluster 2 mean for Cash Ratio: 33.57142857142857
Cluster 3 mean for Cash Ratio: 70.7095709570957
Cluster 4 mean for Cash Ratio: 130.5

****** Feature Net Cash Flow ******
Cluster 0 mean for Net Cash Flow: -4070833333.3333335
Cluster 1 mean for Net Cash Flow: 6140875.0
Cluster 2 mean for Net Cash Flow: -568400000.0
Cluster 3 mean for Net Cash Flow: 125750287.12871288
Cluster 4 mean for Net Cash Flow: 16755500000.0

****** Feature Net Income ******
Cluster 0 mean for Net Income: 11045666666.666666
Cluster 1 mean for Net Income: -603718000.0
Cluster 2 mean for Net Income: -4968157142.857142
Cluster 3 mean for Net Income: 1295945059.4059405
Cluster 4 mean for Net Income: 13654000000.0

****** Feature Earnings Per Share ******
Cluster 0 mean for Earnings Per Share: 2.03
Cluster 1 mean for Earnings Per Share: -3.2075
Cluster 2 mean for Earnings Per Share: -10.84142857142857
Cluster 3 mean for Earnings Per Share: 3.433415841584158
Cluster 4 mean for Earnings Per Share: 3.295

****** Feature Estimated Shares Outstanding ******
Cluster 0 mean for Estimated Shares Outstanding: 3507085088.125
Cluster 1 mean for Estimated Shares Outstanding: 547905122.985625
Cluster 2 mean for Estimated Shares Outstanding: 398169036.44285715
Cluster 3 mean for Estimated Shares Outstanding: 452037260.1709571
Cluster 4 mean for Estimated Shares Outstanding: 2791829362.1

****** Feature P/E Ratio ******
Cluster 0 mean for P/E Ratio: 26.986866739666667
Cluster 1 mean for P/E Ratio: 39.2437444469375
Cluster 2 mean for P/E Ratio: 42.28454148142857
Cluster 3 mean for P/E Ratio: 32.38692420680858
Cluster 4 mean for P/E Ratio: 13.649695574999999

****** Feature P/B Ratio ******
Cluster 0 mean for P/B Ratio: -6.238311734833332
Cluster 1 mean for P/B Ratio: 0.6717558973125
Cluster 2 mean for P/B Ratio: -11.589501539285711
Cluster 3 mean for P/B Ratio: -1.4586917297656765
Cluster 4 mean for P/B Ratio: 1.5084839159999999


# Merge individual dataframes and write comparisions to new table

df_all_clusters = [df_kmean_cluster_0, df_kmean_cluster_1, df_kmean_cluster_2, df_kmean_cluster_3, df_kmean_cluster_4]

mean_table = pd.DataFrame()

for idx, df in enumerate(df_all_clusters):
    temp_mean_dict = {}

    for col in num_cols:
        if pd.api.types.is_numeric_dtype(df[col]):
            other_dfs = [df_all_clusters[i] for i in range(len(df_all_clusters)) if i != idx]  # Exclude the current dataframe
            col_means = [other_df[col].mean() for other_df in other_dfs]
            temp_mean_dict[col] = sum(col_means) / len(col_means)
        else:
            temp_mean_dict[col] = 'Non-numeric'

    # Add results
    temp_mean_df = pd.DataFrame([temp_mean_dict], index=[f'DF {idx}'])
    mean_table = pd.concat([mean_table, temp_mean_df])

# Set pandas option to display float values with 6 decimal places
pd.options.display.float_format = '{:,.2f}'.format

mean_table
#import ace_tools as tools; tools.display_dataframe_to_user(name="Mean Table from Other DataFrames", dataframe=mean_table)


# Loop through the selected columns and plot boxplots in two columns

n_cols = 2

# Iterate through numberical columns
for i, col in enumerate(list_numcol):
    if col in df_kmean.columns:

        # Set up subplots
        fig, axs = plt.subplots(1, n_cols, figsize=(16, 5))

        # Create boxplot
        sns.boxplot(data=df_kmean, x='Cluster', y=col, ax=axs[0], hue='Cluster', palette='viridis')
        axs[0].set_title(f'Boxplot of {col} by Cluster')
        axs[0].set_xticklabels(axs[0].get_xticklabels(), rotation=90)

        if i + 1 < len(list_numcol):
            next_col = list_numcol[i + 1]
            if next_col in df_kmean.columns:
                sns.boxplot(data=df_kmean, x='Cluster', y=next_col, ax=axs[1], hue='Cluster', palette='viridis')
                axs[1].set_title(f'Boxplot of {next_col} by Cluster')
                axs[1].set_xticklabels(axs[1].get_xticklabels(), rotation=90)
        else:
            axs[1].axis('off')

        plt.tight_layout()
        plt.show()

        if i + 1 < len(list_numcol):
            i += 1


# Show the means
df_means_by_cluster = df_kmean.groupby('Cluster')[list_numcol].mean()
df_means_by_cluster = df_means_by_cluster.T  # Transpose to get clusters as columns
df_means_by_cluster


# Plot boxplots with mean
for col in list_numcol:
    if col in df_kmean.columns:
        plt.figure(figsize=(8, 5))
        sns.boxplot(data=df_kmean, x='Cluster', y=col, hue='Cluster', palette='viridis', showmeans=False)

        # Use an overlaid  pointplot to show means
        sns.pointplot(data=df_kmean, x='Cluster', y=col, hue='Cluster', dodge=True, markers='D',
                      scale=0.75, ci=None, palette='dark', legend=False)

        plt.xticks(rotation=90)
        plt.title(f'Boxplot of {col} by Cluster')
        plt.show()


# Re-display final results from our k-means analysis
mean_table


# @title
# Show pivot of sectors and subsectors with counts
df_kmean_cluster_0_pivot = df_kmean_cluster_0.pivot_table(
    index=['GICS Sector', 'GICS Sub Industry'],
    values='Security',
    aggfunc={'Security': 'count'},
    fill_value=0
).reset_index()

# Rename columns
df_kmean_cluster_0_pivot.columns = ['GICS Sector', 'GICS Sub Industry', 'Company Count']

# Calculate grand total
grand_total_count = df_kmean_cluster_0_pivot['Company Count'].sum()
grand_total_df = pd.DataFrame({
    'GICS Sector': ['Grand Total'],
    'GICS Sub Industry': [''],
    'Company Count': [grand_total_count]
})

# Combine counts with grand total
combined = pd.concat([df_kmean_cluster_0_pivot, grand_total_df], ignore_index=True)

# Display the combined DataFrame
print('Cluster 0: Sector, Sub Industry and Company Count')
combined

Cluster 0: Sector, Sub Industry and Company Count


# @title
# Show pivot of sectors and subsectors with counts
df_kmean_cluster_1_pivot = df_kmean_cluster_1.pivot_table(
    index=['GICS Sector', 'GICS Sub Industry'],
    values='Security',
    aggfunc={'Security': 'count'},
    fill_value=0
).reset_index()

# Rename columns
df_kmean_cluster_1_pivot.columns = ['GICS Sector', 'GICS Sub Industry', 'Company Count']

# Calculate grand total
grand_total_count = df_kmean_cluster_1_pivot['Company Count'].sum()
grand_total_df = pd.DataFrame({
    'GICS Sector': ['Grand Total'],
    'GICS Sub Industry': [''],
    'Company Count': [grand_total_count]
})

# Combine counts with grand total
combined = pd.concat([df_kmean_cluster_1_pivot, grand_total_df], ignore_index=True)

# Display the combined DataFrame
print('Cluster 1: Sector, Sub Industry and Company Count')
combined

Cluster 1: Sector, Sub Industry and Company Count


# @title
# Show pivot of sectors and subsectors with counts
df_kmean_cluster_2_pivot = df_kmean_cluster_2.pivot_table(
    index=['GICS Sector', 'GICS Sub Industry'],
    values='Security',
    aggfunc={'Security': 'count'},
    fill_value=0
).reset_index()

# Rename columns
df_kmean_cluster_2_pivot.columns = ['GICS Sector', 'GICS Sub Industry', 'Company Count']

# Calculate grand total
grand_total_count = df_kmean_cluster_2_pivot['Company Count'].sum()
grand_total_df = pd.DataFrame({
    'GICS Sector': ['Grand Total'],
    'GICS Sub Industry': [''],
    'Company Count': [grand_total_count]
})

# Combine counts with grand total
combined = pd.concat([df_kmean_cluster_2_pivot, grand_total_df], ignore_index=True)

# Display the combined DataFrame
print('Cluster 2: Sector, Sub Industry and Company Count')
combined

Cluster 2: Sector, Sub Industry and Company Count


# @title
# Show all rows in the DataFrame
pd.set_option('display.max_rows', None)

# Show pivot of sectors and subsectors with counts
df_kmean_cluster_3_pivot = df_kmean_cluster_3.pivot_table(
    index=['GICS Sector', 'GICS Sub Industry'],
    values='Security',
    aggfunc={'Security': 'count'},
    fill_value=0
).reset_index()

# Rename columns
df_kmean_cluster_3_pivot.columns = ['GICS Sector', 'GICS Sub Industry', 'Company Count']

# Calculate grand total
grand_total_count = df_kmean_cluster_3_pivot['Company Count'].sum()
grand_total_df = pd.DataFrame({
    'GICS Sector': ['Grand Total'],
    'GICS Sub Industry': [''],
    'Company Count': [grand_total_count]
})

# Combine counts with grand total
combined = pd.concat([df_kmean_cluster_3_pivot, grand_total_df], ignore_index=True)

# Display the combined DataFrame
print('Cluster 3: Sector, Sub Industry and Company Count')
combined

Cluster 3: Sector, Sub Industry and Company Count


# @title
# Show pivot of sectors and subsectors with counts
df_kmean_cluster_4_pivot = df_kmean_cluster_4.pivot_table(
    index=['GICS Sector', 'GICS Sub Industry'],
    values='Security',
    aggfunc={'Security': 'count'},
    fill_value=0
).reset_index()

# Rename columns
df_kmean_cluster_4_pivot.columns = ['GICS Sector', 'GICS Sub Industry', 'Company Count']

# Calculate grand total
grand_total_count = df_kmean_cluster_4_pivot['Company Count'].sum()
grand_total_df = pd.DataFrame({
    'GICS Sector': ['Grand Total'],
    'GICS Sub Industry': [''],
    'Company Count': [grand_total_count]
})

# Combine counts with grand total
combined = pd.concat([df_kmean_cluster_4_pivot, grand_total_df], ignore_index=True)

# Display the combined DataFrame
print('Cluster 4: Sector, Sub Industry and Company Count')
combined

Cluster 4: Sector, Sub Industry and Company Count


!pip install nbconvert

Requirement already satisfied: nbconvert in /usr/local/lib/python3.10/dist-packages (6.5.4)
Requirement already satisfied: lxml in /usr/local/lib/python3.10/dist-packages (from nbconvert) (4.9.4)
Requirement already satisfied: beautifulsoup4 in /usr/local/lib/python3.10/dist-packages (from nbconvert) (4.12.3)
Requirement already satisfied: bleach in /usr/local/lib/python3.10/dist-packages (from nbconvert) (6.1.0)
Requirement already satisfied: defusedxml in /usr/local/lib/python3.10/dist-packages (from nbconvert) (0.7.1)
Requirement already satisfied: entrypoints>=0.2.2 in /usr/local/lib/python3.10/dist-packages (from nbconvert) (0.4)
Requirement already satisfied: jinja2>=3.0 in /usr/local/lib/python3.10/dist-packages (from nbconvert) (3.1.4)
Requirement already satisfied: jupyter-core>=4.7 in /usr/local/lib/python3.10/dist-packages (from nbconvert) (5.7.2)
Requirement already satisfied: jupyterlab-pygments in /usr/local/lib/python3.10/dist-packages (from nbconvert) (0.3.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from nbconvert) (2.1.5)
Requirement already satisfied: mistune<2,>=0.8.1 in /usr/local/lib/python3.10/dist-packages (from nbconvert) (0.8.4)
Requirement already satisfied: nbclient>=0.5.0 in /usr/local/lib/python3.10/dist-packages (from nbconvert) (0.10.0)
Requirement already satisfied: nbformat>=5.1 in /usr/local/lib/python3.10/dist-packages (from nbconvert) (5.10.4)
Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from nbconvert) (24.1)
Requirement already satisfied: pandocfilters>=1.4.1 in /usr/local/lib/python3.10/dist-packages (from nbconvert) (1.5.1)
Requirement already satisfied: pygments>=2.4.1 in /usr/local/lib/python3.10/dist-packages (from nbconvert) (2.18.0)
Requirement already satisfied: tinycss2 in /usr/local/lib/python3.10/dist-packages (from nbconvert) (1.3.0)
Requirement already satisfied: traitlets>=5.0 in /usr/local/lib/python3.10/dist-packages (from nbconvert) (5.7.1)
Requirement already satisfied: platformdirs>=2.5 in /usr/local/lib/python3.10/dist-packages (from jupyter-core>=4.7->nbconvert) (4.3.6)
Requirement already satisfied: jupyter-client>=6.1.12 in /usr/local/lib/python3.10/dist-packages (from nbclient>=0.5.0->nbconvert) (6.1.12)
Requirement already satisfied: fastjsonschema>=2.15 in /usr/local/lib/python3.10/dist-packages (from nbformat>=5.1->nbconvert) (2.20.0)
Requirement already satisfied: jsonschema>=2.6 in /usr/local/lib/python3.10/dist-packages (from nbformat>=5.1->nbconvert) (4.23.0)
Requirement already satisfied: soupsieve>1.2 in /usr/local/lib/python3.10/dist-packages (from beautifulsoup4->nbconvert) (2.6)
Requirement already satisfied: six>=1.9.0 in /usr/local/lib/python3.10/dist-packages (from bleach->nbconvert) (1.16.0)
Requirement already satisfied: webencodings in /usr/local/lib/python3.10/dist-packages (from bleach->nbconvert) (0.5.1)
Requirement already satisfied: attrs>=22.2.0 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=2.6->nbformat>=5.1->nbconvert) (24.2.0)
Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=2.6->nbformat>=5.1->nbconvert) (2023.12.1)
Requirement already satisfied: referencing>=0.28.4 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=2.6->nbformat>=5.1->nbconvert) (0.35.1)
Requirement already satisfied: rpds-py>=0.7.1 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=2.6->nbformat>=5.1->nbconvert) (0.20.0)
Requirement already satisfied: pyzmq>=13 in /usr/local/lib/python3.10/dist-packages (from jupyter-client>=6.1.12->nbclient>=0.5.0->nbconvert) (24.0.1)
Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.10/dist-packages (from jupyter-client>=6.1.12->nbclient>=0.5.0->nbconvert) (2.8.2)
Requirement already satisfied: tornado>=4.1 in /usr/local/lib/python3.10/dist-packages (from jupyter-client>=6.1.12->nbclient>=0.5.0->nbconvert) (6.3.3)


# Create an HTML version of this notebook - do this last
%%shell
jupyter nbconvert --to html '/content/drive/MyDrive/Learning/Data Coursework/PGP-DSBA/7-Unsupervised Learning/Project 7/USL_Project_LearnerNotebook_FullCode_Tuesday_Frase_vFinal.ipynb'

[NbConvertApp] Converting notebook /content/drive/MyDrive/Learning/Data Coursework/PGP-DSBA/7-Unsupervised Learning/Project 7/USL_Project_LearnerNotebook_FullCode_Tuesday_Frase_vFinal.ipynb to html
[NbConvertApp] Writing 11319755 bytes to /content/drive/MyDrive/Learning/Data Coursework/PGP-DSBA/7-Unsupervised Learning/Project 7/USL_Project_LearnerNotebook_FullCode_Tuesday_Frase_vFinal.html

	Ticker Symbol	Security	GICS Sector	GICS Sub Industry	Current Price	Price Change	Volatility	ROE	Cash Ratio	Net Cash Flow	Net Income	Earnings Per Share	Estimated Shares Outstanding	P/E Ratio	P/B Ratio
102	DVN	Devon Energy Corp.	Energy	Oil & Gas Exploration & Production	32.000000	-15.478079	2.923698	205	70	830000000	-14454000000	-35.55	4.065823e+08	93.089287	1.785616
125	FB	Facebook	Information Technology	Internet Software & Services	104.660004	16.224320	1.320606	8	958	592000000	3669000000	1.31	2.800763e+09	79.893133	5.884467
11	AIV	Apartment Investment & Mgmt	Real Estate	REITs	40.029999	7.578608	1.163334	15	47	21818000	248710000	1.52	1.636250e+08	26.335526	-1.269332
248	PG	Procter & Gamble	Consumer Staples	Personal Products	79.410004	10.660538	0.806056	17	129	160383000	636056000	3.28	4.913916e+08	24.070121	-2.256747
238	OXY	Occidental Petroleum	Energy	Oil & Gas Exploration & Production	67.610001	0.865287	1.589520	32	64	-588000000	-7829000000	-10.23	7.652981e+08	93.089287	3.345102
336	YUM	Yum! Brands Inc	Consumer Discretionary	Restaurants	52.516175	-8.698917	1.478877	142	27	159000000	1293000000	2.97	4.353535e+08	17.682214	-3.838260
112	EQT	EQT Corporation	Energy	Oil & Gas Exploration & Production	52.130001	-21.253771	2.364883	2	201	523803000	85171000	0.56	1.520911e+08	93.089287	9.567952
147	HAL	Halliburton Co.	Energy	Oil & Gas Equipment & Services	34.040001	-5.101751	1.966062	4	189	7786000000	-671000000	-0.79	8.493671e+08	93.089287	17.345857
89	DFS	Discover Financial Services	Financials	Consumer Finance	53.619999	3.653584	1.159897	20	99	2288000000	2297000000	5.14	4.468872e+08	10.431906	-0.375934
173	IVZ	Invesco Ltd.	Financials	Asset Management & Custody Banks	33.480000	7.067477	1.580839	12	67	412000000	968100000	2.26	4.283628e+08	14.814159	4.218620

	Feature	Mean	Median	Min	Max	Range	Standard Deviation	Variance
0	Current Price	80.862345	59.705000	4.500000	1274.949951	1270.449951	98.055086	9614.799896
1	Price Change	4.078194	4.819505	-47.129693	55.051683	102.181377	12.006338	144.152149
2	Volatility	1.525976	1.385593	0.733163	4.580042	3.846879	0.591798	0.350225
3	ROE	39.597059	15.000000	1.000000	917.000000	916.000000	96.547538	9321.427130
4	Cash Ratio	70.023529	47.000000	0.000000	958.000000	958.000000	90.421331	8176.017144
5	Net Cash Flow	55537620.588235	2098000.000000	-11208000000.000000	20764000000.000000	31972000000.000000	1946365312.175789	3788337928441157632.000000
6	Net Income	1494384602.941176	707336000.000000	-23528000000.000000	24442000000.000000	47970000000.000000	3940150279.327937	15524784223688019968.000000
7	Earnings Per Share	2.776662	2.895000	-61.200000	50.090000	111.290000	6.587779	43.398830
8	Estimated Shares Outstanding	577028337.754030	309675137.800000	27672156.860000	6159292035.000000	6131619878.140000	845849595.417695	715461538068278272.000000
9	P/E Ratio	32.612563	20.819876	2.935451	528.039074	525.103623	44.348731	1966.809957
10	P/B Ratio	-1.718249	-1.067170	-76.119077	129.064585	205.183663	13.966912	195.074625

	Category	Count	Percentage
93	Electronic Equipment & Instruments	1	0.294118
85	Financial Exchanges & Data	1	0.294118
86	Water Utilities	1	0.294118
87	Home Entertainment Software	1	0.294118
88	Drug Retail	1	0.294118
89	Electrical Components & Equipment	1	0.294118
90	Semiconductor Equipment	1	0.294118
91	Multi-line Insurance	1	0.294118
92	Copper	1	0.294118
96	Tires & Rubber	1	0.294118
94	Diversified Commercial Services	1	0.294118
95	Consumer Electronics	1	0.294118
83	Agricultural Products	1	0.294118
97	Industrial Materials	1	0.294118
98	Motorcycle Manufacturers	1	0.294118
99	Technology Hardware, Storage & Peripherals	1	0.294118
100	Real Estate Services	1	0.294118
101	Trucking	1	0.294118
102	Networking Equipment	1	0.294118
84	Metal & Glass Containers	1	0.294118
79	Multi-Sector Holdings	1	0.294118
82	Distributors	1	0.294118
70	Human Resource & Employment Services	1	0.294118
62	Gold	1	0.294118
63	Environmental Services	1	0.294118
64	Home Furnishings	1	0.294118
65	Household Appliances	1	0.294118
66	Apparel, Accessories & Luxury Goods	1	0.294118
67	Specialty Retail	1	0.294118
68	Life Sciences Tools & Services	1	0.294118
69	Publishing	1	0.294118
71	Steel	1	0.294118
81	Computer Hardware	1	0.294118
72	Housewares & Specialties	1	0.294118
73	Thrifts & Mortgage Finance	1	0.294118
74	Brewers	1	0.294118
75	Technology, Hardware, Software and Supplies	1	0.294118
76	Personal Products	1	0.294118
77	Industrial Gases	1	0.294118
78	Office REITs	1	0.294118
80	Alternative Carriers	1	0.294118
103	Casinos & Gaming	1	0.294118

	GICS Sector	GICS Sector Int	Current Price	Price Change	Volatility	ROE	Cash Ratio	Net Cash Flow	Net Income	Earnings Per Share	Estimated Shares Outstanding	P/E Ratio	P/B Ratio
0	Industrials	6	0.029793	0.559101	0.247990	0.146288	0.053236	0.331665	0.649114	0.652260	0.104452	0.001491	0.328169
1	Health Care	5	0.043087	0.542850	0.380757	0.140830	0.080376	0.352152	0.597707	0.578219	0.261814	0.030224	0.328335
2	Health Care	5	0.031808	0.571834	0.140499	0.021834	0.069937	0.379895	0.582677	0.576332	0.240842	0.023500	0.369059
3	Information Technology	7	0.070400	0.598024	0.162344	0.008734	0.187891	0.343024	0.503597	0.561236	0.076973	0.136392	0.391448
4	Information Technology	7	0.040002	0.443347	0.251634	0.014192	0.283925	0.360413	0.505001	0.552700	0.362110	0.334251	0.376145

	GICS Sector	Mean Price Change	Min	Max	Range
0	Health Care	9.585652	-12.532337	33.177346	45.709684
1	Consumer Staples	8.684750	-12.017268	24.496225	36.513493
2	Information Technology	7.217476	-23.790903	55.051683	78.842586
3	Telecommunications Services	6.956980	-2.301255	24.707504	27.008759
4	Real Estate	6.205548	-13.067267	15.573900	28.641168
5	Consumer Discretionary	5.846093	-33.131268	34.803917	67.935185
6	Materials	5.589738	-31.685167	37.489677	69.174843
7	Financials	3.865406	-14.292764	15.462833	29.755597
8	Industrials	2.833127	-23.244191	20.432767	43.676958
9	Utilities	0.803657	-8.230553	8.596874	16.827427
10	Energy	-10.228289	-47.129693	17.341522	64.471216

	count	unique	top	freq	mean	std	min	25%	50%	75%	max
Ticker Symbol	340	340	AAL	1	NaN	NaN	NaN	NaN	NaN	NaN	NaN
Security	340	340	American Airlines Group	1	NaN	NaN	NaN	NaN	NaN	NaN	NaN
GICS Sector	340	11	Industrials	53	NaN	NaN	NaN	NaN	NaN	NaN	NaN
GICS Sub Industry	340	104	Oil & Gas Exploration & Production	16	NaN	NaN	NaN	NaN	NaN	NaN	NaN
Current Price	340.0	NaN	NaN	NaN	80.862345	98.055086	4.5	38.555	59.705	92.880001	1274.949951
Price Change	340.0	NaN	NaN	NaN	4.078194	12.006338	-47.129693	-0.939484	4.819505	10.695493	55.051683
Volatility	340.0	NaN	NaN	NaN	1.525976	0.591798	0.733163	1.134878	1.385593	1.695549	4.580042
ROE	340.0	NaN	NaN	NaN	39.597059	96.547538	1.0	9.75	15.0	27.0	917.0
Cash Ratio	340.0	NaN	NaN	NaN	70.023529	90.421331	0.0	18.0	47.0	99.0	958.0
Net Cash Flow	340.0	NaN	NaN	NaN	55537620.588235	1946365312.175789	-11208000000.0	-193906500.0	2098000.0	169810750.0	20764000000.0
Net Income	340.0	NaN	NaN	NaN	1494384602.941176	3940150279.327937	-23528000000.0	352301250.0	707336000.0	1899000000.0	24442000000.0
Earnings Per Share	340.0	NaN	NaN	NaN	2.776662	6.587779	-61.2	1.5575	2.895	4.62	50.09
Estimated Shares Outstanding	340.0	NaN	NaN	NaN	577028337.75403	845849595.417695	27672156.86	158848216.1	309675137.8	573117457.325	6159292035.0
P/E Ratio	340.0	NaN	NaN	NaN	32.612563	44.348731	2.935451	15.044653	20.819876	31.764755	528.039074
P/B Ratio	340.0	NaN	NaN	NaN	-1.718249	13.966912	-76.119077	-4.352056	-1.06717	3.917066	129.064585

	Category	Count	Percentage
0	Industrials	53	15.588235
1	Financials	49	14.411765
2	Health Care	40	11.764706
3	Consumer Discretionary	40	11.764706
4	Information Technology	33	9.705882
5	Energy	30	8.823529
6	Real Estate	27	7.941176
7	Utilities	24	7.058824
8	Materials	20	5.882353
9	Consumer Staples	19	5.588235
10	Telecommunications Services	5	1.470588

	Category	Count	Percentage
0	Oil & Gas Exploration & Production	16	4.705882
2	Industrial Conglomerates	14	4.117647
1	REITs	14	4.117647
3	Electric Utilities	12	3.529412
4	Internet Software & Services	12	3.529412
5	Health Care Equipment	11	3.235294
6	MultiUtilities	11	3.235294
7	Banks	10	2.941176
8	Property & Casualty Insurance	8	2.352941
9	Diversified Financial Services	7	2.058824

	GICS Sector	Count	Average Current Price	Percentage of Total
4	Health Care	40	132.048250	11.764706
0	Consumer Discretionary	40	128.095404	11.764706
8	Real Estate	27	90.976925	7.941176
7	Materials	20	76.551501	5.882353
5	Industrials	53	74.411807	15.588235
1	Consumer Staples	19	71.972807	5.588235
6	Information Technology	33	63.548485	9.705882
3	Financials	49	58.659183	14.411765
10	Utilities	24	52.968542	7.058824
2	Energy	30	46.042334	8.823529
9	Telecommunications Services	5	32.964000	1.470588

	Cash Ratio
GICS Sector
Information Technology	149.818182
Telecommunications Services	117.000000
Health Care	103.775000
Financials	98.591837
Consumer Staples	70.947368
Energy	51.133333
Real Estate	50.111111
Consumer Discretionary	49.575000
Materials	41.700000
Industrials	36.188679
Utilities	13.625000

	P/E Ratio
GICS Sector
Energy	72.897709
Information Technology	43.782546
Real Estate	43.065585
Health Care	41.135272
Consumer Discretionary	35.211613
Consumer Staples	25.521195
Materials	24.585352
Utilities	18.719412
Industrials	18.259380
Financials	16.023151
Telecommunications Services	12.222578

	Current Price	Price Change	Volatility	ROE	Cash Ratio	Net Cash Flow	Net Income	Earnings Per Share	Estimated Shares Outstanding	P/E Ratio	P/B Ratio
DF 0	64.38	4.33	1.69	210.96	74.73	4,079,747,790.53	2,344,517,479.14	-1.83	1,047,485,195.42	31.89	-2.72
DF 1	60.51	6.18	1.48	172.83	76.63	3,060,504,238.45	5,256,863,645.80	-0.52	1,787,280,186.71	28.83	-4.44
DF 2	55.47	4.69	1.55	59.06	84.27	3,204,139,457.20	6,347,973,431.52	1.39	1,824,714,208.35	28.07	-1.38
DF 3	55.74	4.58	1.64	212.64	74.99	3,030,601,885.42	4,781,947,880.95	-2.18	1,811,247,152.41	30.54	-3.91
DF 4	70.15	2.85	1.68	214.33	60.04	-1,126,835,542.80	1,692,434,145.80	-2.15	1,226,299,126.93	35.23	-4.65

Cluster	0	1	2	3	4
Current Price	48.73	64.21	84.36	83.30	25.64
Price Change	5.32	-2.10	3.85	4.31	11.24
Volatility	1.27	2.12	1.83	1.50	1.32
ROE	26.00	178.50	633.57	19.26	12.50
Cash Ratio	71.75	64.12	33.57	70.71	130.50
Net Cash Flow	-4,070,833,333.33	6,140,875.00	-568,400,000.00	125,750,287.13	16,755,500,000.00
Net Income	11,045,666,666.67	-603,718,000.00	-4,968,157,142.86	1,295,945,059.41	13,654,000,000.00
Earnings Per Share	2.03	-3.21	-10.84	3.43	3.29
Estimated Shares Outstanding	3,507,085,088.12	547,905,122.99	398,169,036.44	452,037,260.17	2,791,829,362.10
P/E Ratio	26.99	39.24	42.28	32.39	13.65
P/B Ratio	-6.24	0.67	-11.59	-1.46	1.51

	GICS Sector	GICS Sub Industry	Company Count
0	Consumer Discretionary	Automobile Manufacturers	1
1	Consumer Staples	Soft Drinks	1
2	Energy	Integrated Oil & Gas	1
3	Energy	Oil & Gas Exploration & Production	1
4	Financials	Banks	3
5	Health Care	Health Care Distributors	1
6	Health Care	Pharmaceuticals	1
7	Information Technology	Internet Software & Services	1
8	Telecommunications Services	Integrated Telecommunications Services	2
9	Grand Total		12

	GICS Sector	GICS Sub Industry	Company Count
0	Consumer Discretionary	Cable & Satellite	1
1	Consumer Staples	Household Products	2
2	Energy	Integrated Oil & Gas	1
3	Energy	Oil & Gas Exploration & Production	1
4	Financials	Diversified Financial Services	1
5	Industrials	Building Products	1
6	Grand Total		7

	Current Price	Price Change	Volatility	ROE	Cash Ratio	Net Cash Flow	Net Income	Earnings Per Share	Estimated Shares Outstanding	P/E Ratio	P/B Ratio	GICS Sector Int
0	42.349998	9.999995	1.687151	135	51	-604000000	7610000000	11.390000	668129938.500000	3.718174	-8.784219	6
1	59.240002	8.339433	2.197887	130	77	51000000	5144000000	3.150000	1633015873.000000	18.806350	-8.750068	5
2	44.910000	11.301121	1.273646	21	67	938000000	4423000000	2.940000	1504421769.000000	15.275510	-0.394171	5
3	93.940002	13.977195	1.357679	9	180	-240840000	629551000	1.260000	499643650.800000	74.555557	4.199651	7
4	55.320000	-1.827858	1.701169	14	272	315120000	696878000	0.310000	2247993548.000000	178.451613	1.059810	7
5	36.680000	-12.017268	1.516493	10	49	-189000000	1849000000	2.990000	618394648.800000	12.267559	7.496831	2
6	276.570007	6.189286	1.116976	30	25	90885000	596541000	8.910000	66951851.850000	31.040405	129.064585	7
7	43.230000	2.174424	1.124186	9	14	287000000	636000000	2.600000	244615384.600000	16.626923	-0.719497	11
8	58.270000	2.371753	1.068485	11	9	13900000	2052300000	3.130000	421897810.200000	18.456543	-3.022649	11
9	59.900002	3.027181	1.048295	14	99	-308000000	2533000000	5.880000	430782312.900000	10.187075	-1.883912	4
10	61.970001	8.358108	1.106965	2	99	-129000000	2196000000	1.690000	1299408284.000000	36.668640	-4.327138	4
11	40.029999	7.578608	1.163334	15	47	21818000	248710000	1.520000	163625000.000000	26.335526	-1.269332	9
12	80.540001	1.897773	1.112604	3	99	-30351000	141555000	2.080000	68055288.460000	38.721154	-4.072615	4
13	40.939999	-0.606943	1.052050	10	99	166000000	356800000	2.070000	172367149.800000	19.777777	-9.855706	4
14	52.630001	-23.790903	1.384502	10	225	50823000	321406000	1.800000	178558888.900000	29.238889	4.282358	7
15	56.009998	26.461948	1.974323	10	13	-2276034000	334906000	3.010000	111264451.800000	18.607973	-13.649723	8
16	80.510002	2.066436	1.773431	35	74	-34000000	848000000	6.610000	128290469.000000	12.180031	-1.114658	6
17	62.090000	6.592275	1.053266	11	99	-162000000	2171000000	5.120000	424023437.500000	12.126953	-4.327138	4
18	65.919998	13.753230	1.283795	601	45	-90800000	153900000	1.600000	96187500.000000	41.199999	-0.877453	6
19	190.750000	22.338380	2.022921	2	195	66000000	144000000	0.680000	211764705.900000	280.514706	-14.171389	5
20	18.670000	26.834239	1.460030	18	131	1795000000	1377000000	1.130000	1218584071.000000	16.522124	3.857756	7
21	53.590000	2.212474	1.089266	18	37	3390000	590859000	2.460000	240186585.400000	21.784553	-4.490342	6
22	159.759995	-6.611335	2.093065	18	67	13200000	516000000	9.490000	54373024.240000	16.834562	-31.015380	4
23	162.330002	17.163478	1.630259	25	362	413000000	6939000000	9.150000	758360655.700000	17.740984	24.012322	5
24	106.419998	-2.420686	1.222260	22	67	-281000000	1562000000	8.600000	181627907.000000	12.374418	-13.398380	4
25	96.949997	10.233087	1.165804	10	39	7194000	685074000	1.420000	482446478.900000	68.274646	-20.813577	9
26	675.890015	32.268105	1.460386	4	58	1333000000	596000000	1.280000	465625000.000000	528.039074	3.904430	1
27	59.660000	2.350316	1.480914	19	1	-1300000	442600000	3.930000	112620865.100000	15.180662	-7.970104	1
28	139.440002	-0.620053	1.511654	11	70	-38200000	2560000000	9.730000	263103802.700000	14.330935	-31.006773	5
29	92.209999	3.910301	1.105032	23	99	10000000	1385000000	4.930000	280933062.900000	18.703854	-7.759856	4
30	44.470001	11.397804	2.405408	917	80	698000000	-23528000000	-61.200000	384444444.400000	93.089287	4.970809	3
31	48.580002	-20.802083	2.435165	52	22	-6430000000	-6692000000	-13.180000	507738998.500000	93.089287	-12.860938	3
32	52.230000	2.693667	1.007762	24	175	768300000	763500000	2.470000	309109311.700000	21.145749	8.202923	7
33	7.398807	1.647784	2.592065	3	37	42000000	-322000000	-0.310000	1038709677.000000	18.687607	2.639814	6
34	38.709999	23.319529	1.886335	11	70	-3025000000	892000000	1.210000	737190082.600000	31.991735	0.290291	7
35	184.130005	4.857630	1.132875	8	47	-108953000	741733000	5.540000	133886823.100000	33.236463	-3.089477	9
36	145.149994	17.902683	1.847180	29	163	218000000	1364000000	2.860000	1044055944.000000	18.249407	3.954975	7
37	59.750000	8.596874	1.171525	9	4	22000000	476000000	2.660000	178947368.400000	22.462406	-4.895294	11
38	69.550003	-6.216290	0.900066	25	99	474000000	5163000000	3.900000	506660363.000000	10.263506	-0.609074	4
39	144.589996	10.105078	1.155905	82	24	-431000000	5176000000	7.520000	688297872.300000	19.227393	22.032612	6

	GICS Sector	GICS Sector Int	Current Price	Price Change	Volatility	ROE	Cash Ratio	Net Cash Flow	Net Income	Earnings Per Share	Estimated Shares Outstanding	P/E Ratio	P/B Ratio
0	Industrials	6	0.029793	0.559101	0.247990	0.146288	0.053236	0.331665	0.649114	0.652260	0.104452	0.001491	0.328169
1	Health Care	5	0.043087	0.542850	0.380757	0.140830	0.080376	0.352152	0.597707	0.578219	0.261814	0.030224	0.328335
2	Health Care	5	0.031808	0.571834	0.140499	0.021834	0.069937	0.379895	0.582677	0.576332	0.240842	0.023500	0.369059
3	Information Technology	7	0.070400	0.598024	0.162344	0.008734	0.187891	0.343024	0.503597	0.561236	0.076973	0.136392	0.391448
4	Information Technology	7	0.040002	0.443347	0.251634	0.014192	0.283925	0.360413	0.505001	0.552700	0.362110	0.334251	0.376145
5	Consumer Staples	2	0.025330	0.343628	0.203627	0.009825	0.051148	0.344645	0.529018	0.576781	0.096340	0.017772	0.407517
6	Information Technology	7	0.214152	0.521807	0.099773	0.031659	0.026096	0.353399	0.502909	0.629976	0.006406	0.053523	1.000000
7	Utilities	11	0.030485	0.482516	0.101647	0.008734	0.014614	0.359533	0.503731	0.573277	0.035381	0.026074	0.367474
8	Utilities	11	0.042324	0.484447	0.087167	0.010917	0.009395	0.350991	0.533256	0.578039	0.064294	0.029558	0.356249
9	Financials	4	0.043607	0.490861	0.081919	0.014192	0.103340	0.340923	0.543277	0.602750	0.065743	0.013810	0.361799
10	Financials	4	0.045236	0.543032	0.097170	0.001092	0.103340	0.346522	0.536252	0.565100	0.207406	0.064241	0.349891
11	Real Estate	9	0.027966	0.535404	0.111823	0.015284	0.049061	0.351239	0.495658	0.563573	0.022172	0.044563	0.364794
12	Financials	4	0.059853	0.479808	0.098636	0.002183	0.103340	0.349607	0.493424	0.568605	0.006586	0.068150	0.351132
13	Financials	4	0.028683	0.455296	0.082895	0.009825	0.103340	0.355749	0.497911	0.568515	0.023598	0.032074	0.322947
14	Information Technology	7	0.037884	0.228406	0.169316	0.009825	0.234864	0.352146	0.497173	0.566089	0.024608	0.050092	0.391851
15	Materials	8	0.040545	0.720206	0.322641	0.009825	0.013570	0.279368	0.497455	0.576961	0.013633	0.029847	0.304456
16	Industrials	6	0.059829	0.481459	0.270419	0.037118	0.077244	0.349493	0.508151	0.609309	0.016410	0.017605	0.365548
17	Financials	4	0.045330	0.525751	0.083211	0.010917	0.103340	0.345490	0.535731	0.595921	0.064641	0.017504	0.349891
18	Industrials	6	0.048345	0.595832	0.143137	0.655022	0.046973	0.347717	0.493681	0.564291	0.011174	0.072870	0.366704
19	Health Care	5	0.146602	0.679851	0.335274	0.001092	0.203549	0.352621	0.493475	0.556025	0.030023	0.528618	0.301913
20	Information Technology	7	0.011154	0.723849	0.188950	0.018559	0.136743	0.406700	0.519179	0.560068	0.194225	0.025874	0.389782
21	Industrials	6	0.038640	0.482888	0.092569	0.018559	0.038622	0.350663	0.502790	0.572019	0.034659	0.035896	0.349096
22	Financials	4	0.122209	0.396534	0.353508	0.018559	0.069937	0.350970	0.501230	0.635187	0.004355	0.026469	0.219821
23	Health Care	5	0.124232	0.629206	0.233201	0.026201	0.377871	0.363474	0.635126	0.632132	0.119167	0.028195	0.488009
24	Financials	4	0.080224	0.437546	0.127141	0.022926	0.069937	0.341768	0.523035	0.627190	0.025108	0.017975	0.305681
25	Real Estate	9	0.072769	0.561382	0.112465	0.009825	0.040710	0.350782	0.504755	0.562674	0.074169	0.124431	0.269541
26	Consumer Discretionary	1	0.528466	0.777028	0.189042	0.003275	0.060543	0.392249	0.502898	0.561416	0.071425	1.000000	0.390009
27	Consumer Discretionary	1	0.043418	0.484237	0.194379	0.019651	0.001044	0.350516	0.499700	0.585228	0.013854	0.023320	0.332136
28	Health Care	5	0.106214	0.455167	0.202370	0.010917	0.073069	0.349362	0.543840	0.637344	0.038396	0.021701	0.219863
29	Financials	4	0.069039	0.499504	0.096668	0.024017	0.103340	0.350870	0.519345	0.594213	0.041304	0.030029	0.333161
30	Energy	3	0.031461	0.572780	0.434702	1.000000	0.083507	0.372388	0.000000	0.000000	0.058186	0.171688	0.395206
31	Energy	3	0.034696	0.257656	0.442437	0.055677	0.022965	0.149443	0.350969	0.431485	0.078294	0.171688	0.308300
32	Information Technology	7	0.037569	0.487597	0.071382	0.025109	0.182672	0.374587	0.506389	0.572109	0.045899	0.034679	0.410959
33	Industrials	6	0.002282	0.477362	0.483223	0.002183	0.038622	0.351870	0.483761	0.547129	0.164889	0.029998	0.383846
34	Information Technology	7	0.026927	0.689453	0.299768	0.010917	0.073069	0.255943	0.509068	0.560787	0.115715	0.055334	0.372395
35	Real Estate	9	0.141391	0.508775	0.103905	0.007642	0.049061	0.347149	0.505936	0.599694	0.017322	0.057705	0.355923
36	Information Technology	7	0.110709	0.636441	0.289590	0.030568	0.170146	0.357375	0.518908	0.575613	0.165761	0.029164	0.390255
37	Utilities	11	0.043489	0.545369	0.113953	0.008734	0.004175	0.351245	0.500396	0.573816	0.024671	0.037187	0.347122
38	Financials	4	0.051202	0.400400	0.043387	0.026201	0.103340	0.365382	0.598103	0.584958	0.078118	0.013955	0.368012
39	Industrials	6	0.110268	0.560129	0.109892	0.088428	0.025052	0.337076	0.598374	0.617486	0.107741	0.031026	0.478360

	GICS Sector	GICS Sub Industry	Company Count
0	Consumer Discretionary	Advertising	2
1	Consumer Discretionary	Apparel, Accessories & Luxury Goods	1
2	Consumer Discretionary	Auto Parts & Equipment	2
3	Consumer Discretionary	Automobile Manufacturers	1
4	Consumer Discretionary	Broadcasting & Cable TV	2
5	Consumer Discretionary	Cable & Satellite	2
6	Consumer Discretionary	Consumer Electronics	1
7	Consumer Discretionary	Distributors	1
8	Consumer Discretionary	Home Furnishings	1
9	Consumer Discretionary	Homebuilding	2
10	Consumer Discretionary	Hotels, Resorts & Cruise Lines	4
11	Consumer Discretionary	Household Appliances	1
12	Consumer Discretionary	Housewares & Specialties	1
13	Consumer Discretionary	Internet & Direct Marketing Retail	4
14	Consumer Discretionary	Leisure Products	2
15	Consumer Discretionary	Motorcycle Manufacturers	1
16	Consumer Discretionary	Publishing	1
17	Consumer Discretionary	Restaurants	2
18	Consumer Discretionary	Specialty Retail	1
19	Consumer Discretionary	Specialty Stores	3
20	Consumer Discretionary	Tires & Rubber	1
21	Consumer Staples	Agricultural Products	1
22	Consumer Staples	Brewers	1
23	Consumer Staples	Drug Retail	1
24	Consumer Staples	Household Products	1
25	Consumer Staples	Packaged Foods & Meats	6
26	Consumer Staples	Personal Products	1
27	Consumer Staples	Soft Drinks	3
28	Consumer Staples	Tobacco	1
29	Energy	Integrated Oil & Gas	3
30	Energy	Oil & Gas Equipment & Services	3
31	Energy	Oil & Gas Exploration & Production	10
32	Energy	Oil & Gas Refining & Marketing & Transportation	6
33	Financials	Asset Management & Custody Banks	4
34	Financials	Banks	6
35	Financials	Consumer Finance	5
36	Financials	Diversified Financial Services	5
37	Financials	Financial Exchanges & Data	1
38	Financials	Insurance Brokers	3
39	Financials	Investment Banking & Brokerage	2
40	Financials	Life & Health Insurance	3
41	Financials	Multi-Sector Holdings	1
42	Financials	Multi-line Insurance	1
43	Financials	Property & Casualty Insurance	8
44	Financials	Regional Banks	3
45	Financials	Thrifts & Mortgage Finance	1
46	Health Care	Biotechnology	7
47	Health Care	Health Care Distributors	2
48	Health Care	Health Care Equipment	10
49	Health Care	Health Care Facilities	5
50	Health Care	Health Care Supplies	2
51	Health Care	Life Sciences Tools & Services	1
52	Health Care	Managed Health Care	5
53	Health Care	Pharmaceuticals	4
54	Industrials	Aerospace & Defense	4
55	Industrials	Air Freight & Logistics	2
56	Industrials	Airlines	4
57	Industrials	Building Products	2
58	Industrials	Construction & Farm Machinery & Heavy Trucks	3
59	Industrials	Diversified Commercial Services	1
60	Industrials	Electrical Components & Equipment	1
61	Industrials	Environmental Services	1
62	Industrials	Human Resource & Employment Services	1
63	Industrials	Industrial Conglomerates	14
64	Industrials	Industrial Machinery	5
65	Industrials	Industrial Materials	1
66	Industrials	Railroads	4
67	Industrials	Research & Consulting Services	4
68	Industrials	Trucking	1
69	Information Technology	Application Software	2
70	Information Technology	Computer Hardware	1
71	Information Technology	Data Processing & Outsourced Services	2
72	Information Technology	Electronic Components	2
73	Information Technology	Electronic Equipment & Instruments	1
74	Information Technology	Home Entertainment Software	1
75	Information Technology	IT Consulting & Other Services	3
76	Information Technology	Internet Software & Services	11
77	Information Technology	Networking Equipment	1
78	Information Technology	Semiconductor Equipment	1
79	Information Technology	Semiconductors	5
80	Information Technology	Technology Hardware, Storage & Peripherals	1
81	Materials	Construction Materials	2
82	Materials	Diversified Chemicals	5
83	Materials	Fertilizers & Agricultural Chemicals	2
84	Materials	Gold	1
85	Materials	Industrial Gases	1
86	Materials	Metal & Glass Containers	1
87	Materials	Paper Packaging	2
88	Materials	Specialty Chemicals	3
89	Materials	Steel	1
90	Real Estate	Office REITs	1
91	Real Estate	REITs	14
92	Real Estate	Real Estate Services	1
93	Real Estate	Residential REITs	4
94	Real Estate	Retail REITs	4
95	Real Estate	Specialized REITs	3
96	Telecommunications Services	Alternative Carriers	1
97	Telecommunications Services	Integrated Telecommunications Services	2
98	Utilities	Electric Utilities	12
99	Utilities	MultiUtilities	11
100	Utilities	Water Utilities	1
101	Grand Total		303

Unsupervised Learning: Trade & Ahead¶

Problem Statement¶

Context¶

Objective¶

Data Dictionary¶

Importing necessary libraries and data¶

Data Overview¶

Exploratory Data Analysis (EDA)¶

EDA Helpers¶

Univariate Analysis¶

Bivariate Analysis¶

Correlation Heatmap¶

Scale Data¶

Pairplot¶

Sector vs. Numerical Series¶

Questions to Answer¶

Q1: What does the distribution of stock prices look like?¶

Q2: The stocks of which economic sector have seen the maximum price increase on average?¶

Q3: How are the different variables correlated with each other?¶

Q4: How does the average cash ratio vary across economic sectors?¶

Q5: How does the P/E ratio vary, on average, across economic sectors?¶

Unsupervised Learning Analysis¶

Helpers¶

Data Preprocessing¶

Value Check¶

Outlier Treatment¶

Scaling Treatment¶

Cluster Algorithm Selection¶

K-means Clustering¶

Cluster Profile¶

Hierarchical Clustering¶

Cluster Profile¶

Principal Component Analysis and t-SNE¶

Visualization: PCA with K-Means¶

Visualization: PCA with Hierarchical¶

K-means vs Hierarchical Clustering¶

Final Model¶

Actionable Insights and Recommendations¶

Cluster 0¶

Cluster 1¶

Cluster 2¶

Cluster 3¶

Cluster 4¶

Summary¶

Export Notebook¶